2024-06-11T16:44:59.072 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-11T16:44:59.073 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_lflores@teuthology --verbose --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} --archive /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749704 --name lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi -- /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749704/orig.config.yaml 2024-06-11T16:44:59.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-11T16:44:59.178 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749704 branch: wip-lflores-testing-2-2024-06-10-1925 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: null first_in_suite: false job_id: '7749704' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lflores-testing-2-2024-06-10-1925 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b workunit: branch: wip-lflores-testing-2-2024-06-10-1925 sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b owner: scheduled_lflores@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7590 sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-lflores-testing-2-2024-06-10-1925 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHrc+JvLKqA2yMRCzR8RM8q3JfNQrBNN6oB5r341ncq6X2iYodVzGCSwxysc4T3O6ConBCmBTC+WIMGMM/kx35E= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-10_22:21:08 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-11T16:44:59.178 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa; will attempt to use it 2024-06-11T16:44:59.179 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks 2024-06-11T16:44:59.179 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-11T16:44:59.181 INFO:teuthology.task.internal:Checking packages... 2024-06-11T16:44:59.204 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '37bbe1d8d788b7c9a71280e7d510c295a5853f0b' 2024-06-11T16:44:59.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-11T16:44:59.205 INFO:teuthology.packaging:ref: None 2024-06-11T16:44:59.205 INFO:teuthology.packaging:tag: None 2024-06-11T16:44:59.205 INFO:teuthology.packaging:branch: wip-lflores-testing-2-2024-06-10-1925 2024-06-11T16:44:59.205 INFO:teuthology.packaging:sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T16:44:59.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lflores-testing-2-2024-06-10-1925 2024-06-11T16:44:59.448 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4216-g37bbe1d8-1jammy 2024-06-11T16:44:59.449 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-11T16:44:59.457 INFO:teuthology.task.internal:no buildpackages task found 2024-06-11T16:44:59.458 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-11T16:44:59.469 INFO:teuthology.task.internal:Saving configuration 2024-06-11T16:44:59.478 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-11T16:44:59.484 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-11T16:44:59.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749704', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-11 16:39:53.830224', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKCOuHp7vMcw84uKrooytxTcTTYjEogJhC7C3gAbciKd'} 2024-06-11T16:44:59.508 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-11T16:44:59.515 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-06-11T16:44:59.515 INFO:teuthology.run_tasks:Running task console_log... 2024-06-11T16:44:59.554 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f67d5373be0>, signals=[15]) 2024-06-11T16:44:59.554 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-11T16:44:59.560 INFO:teuthology.task.internal:Opening connections... 2024-06-11T16:44:59.560 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-06-11T16:44:59.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T16:44:59.650 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-11T16:44:59.656 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-06-11T16:44:59.758 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-06-11T16:44:59.759 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Ubuntu" 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="22.04" 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_CODENAME=jammy 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:ID=ubuntu 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE=debian 2024-06-11T16:44:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-11T16:44:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-11T16:44:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-11T16:44:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-11T16:44:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:UBUNTU_CODENAME=jammy 2024-06-11T16:44:59.806 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-06-11T16:44:59.825 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-11T16:44:59.833 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-11T16:44:59.838 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-11T16:44:59.838 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-06-11T16:44:59.852 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-11T16:44:59.857 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-11T16:44:59.858 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-06-11T16:44:59.912 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-11T16:44:59.944 INFO:teuthology.run_tasks:Running task kernel... 2024-06-11T16:44:59.958 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-11T16:44:59.958 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-11T16:44:59.959 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-11T16:44:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T16:44:59.965 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-06-11T16:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-56-generic 2024-06-11T16:45:00.014 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.15.0-56-generic 2024-06-11T16:45:00.014 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-06-11T16:45:00.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-11T16:45:00.311 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-11T16:45:00.402 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-11T16:45:00.474 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-11T16:45:00.652 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,506 kB] 2024-06-11T16:45:00.793 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [478 kB] 2024-06-11T16:45:00.802 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [257 kB] 2024-06-11T16:45:00.806 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-11T16:45:00.807 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-06-11T16:45:00.807 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,919 kB] 2024-06-11T16:45:00.838 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-11T16:45:00.839 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [326 kB] 2024-06-11T16:45:00.845 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-11T16:45:00.845 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [857 kB] 2024-06-11T16:45:00.859 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [606 kB] 2024-06-11T16:45:00.867 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [166 kB] 2024-06-11T16:45:00.869 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-11T16:45:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-11T16:45:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-11T16:45:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-11T16:45:00.898 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-11T16:45:00.980 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,718 kB] 2024-06-11T16:45:01.351 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [639 kB] 2024-06-11T16:45:01.634 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [316 kB] 2024-06-11T16:45:01.672 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-11T16:45:01.686 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-06-11T16:45:01.696 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,975 kB] 2024-06-11T16:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [336 kB] 2024-06-11T16:45:02.447 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-11T16:45:03.040 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,086 kB] 2024-06-11T16:45:03.692 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [706 kB] 2024-06-11T16:45:03.789 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [251 kB] 2024-06-11T16:45:03.803 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-11T16:45:03.804 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.0 kB] 2024-06-11T16:45:03.807 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-06-11T16:45:03.808 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.7 kB] 2024-06-11T16:45:03.808 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-11T16:45:03.809 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-11T16:45:03.812 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-11T16:45:03.816 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-11T16:45:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-11T16:45:04.121 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-06-11T16:45:04.222 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-11T16:45:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.3 kB] 2024-06-11T16:45:04.306 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-11T16:45:09.033 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 14.0 MB in 4s (3,407 kB/s) 2024-06-11T16:45:10.480 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:45:10.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-11T16:45:10.563 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:45:10.774 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-11T16:45:10.774 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-11T16:45:10.960 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-11T16:45:10.960 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-11T16:45:10.960 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-11T16:45:10.961 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-11T16:45:10.961 INFO:teuthology.orchestra.run.smithi012.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-11T16:45:10.961 INFO:teuthology.orchestra.run.smithi012.stdout: python2.7-minimal 2024-06-11T16:45:10.961 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-11T16:45:10.962 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-06-11T16:45:10.962 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-11T16:45:10.962 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-generic linux-image-5.15.0-112-generic 2024-06-11T16:45:10.963 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-5.15.0-112-generic linux-modules-extra-5.15.0-112-generic 2024-06-11T16:45:10.964 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-06-11T16:45:10.964 INFO:teuthology.orchestra.run.smithi012.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-11T16:45:10.964 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-06-11T16:45:10.964 INFO:teuthology.orchestra.run.smithi012.stdout: thermald 2024-06-11T16:45:11.014 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-06-11T16:45:11.014 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-11T16:45:11.014 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.15.0-112-generic linux-modules-5.15.0-112-generic 2024-06-11T16:45:11.014 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-extra-5.15.0-112-generic 2024-06-11T16:45:11.016 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages will be upgraded: 2024-06-11T16:45:11.016 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-11T16:45:11.076 INFO:teuthology.orchestra.run.smithi012.stdout:3 upgraded, 5 newly installed, 0 to remove and 324 not upgraded. 2024-06-11T16:45:11.077 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 113 MB of archives. 2024-06-11T16:45:11.077 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-11T16:45:11.077 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-112-generic amd64 5.15.0-112.122 [22.6 MB] 2024-06-11T16:45:11.532 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-112-generic amd64 5.15.0-112.122 [11.5 MB] 2024-06-11T16:45:11.651 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-112-generic amd64 5.15.0-112.122 [63.9 MB] 2024-06-11T16:45:12.675 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.112.112 [1,692 B] 2024-06-11T16:45:12.730 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.112.112 [2,510 B] 2024-06-11T16:45:12.734 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112 all 5.15.0-112.122 [12.3 MB] 2024-06-11T16:45:13.024 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112-generic amd64 5.15.0-112.122 [2,904 kB] 2024-06-11T16:45:13.097 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.112.112 [2,360 B] 2024-06-11T16:45:13.519 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 113 MB in 2s (54.7 MB/s) 2024-06-11T16:45:13.701 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-5.15.0-112-generic. 2024-06-11T16:45:15.813 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-11T16:45:15.818 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../0-linux-modules-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-11T16:45:16.082 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:18.676 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-image-5.15.0-112-generic. 2024-06-11T16:45:18.697 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../1-linux-image-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-11T16:45:18.842 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:19.337 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-112-generic. 2024-06-11T16:45:19.358 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-11T16:45:19.429 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:25.980 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../3-linux-generic_5.15.0.112.112_amd64.deb ... 2024-06-11T16:45:26.114 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-11T16:45:26.465 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.112.112_amd64.deb ... 2024-06-11T16:45:26.618 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-11T16:45:26.929 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-112. 2024-06-11T16:45:26.952 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../5-linux-headers-5.15.0-112_5.15.0-112.122_all.deb ... 2024-06-11T16:45:26.987 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-11T16:45:32.758 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-112-generic. 2024-06-11T16:45:32.795 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../6-linux-headers-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-11T16:45:32.833 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:37.329 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.112.112_amd64.deb ... 2024-06-11T16:45:37.453 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-11T16:45:37.854 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-11T16:45:37.950 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:38.084 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-generic (5.15.0.112.112) ... 2024-06-11T16:45:38.211 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:39.586 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-11T16:45:39.587 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-11T16:45:39.587 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-112-generic 2024-06-11T16:45:39.587 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-112-generic 2024-06-11T16:45:39.748 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:41.174 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:43.072 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-generic (5.15.0.112.112) ... 2024-06-11T16:45:43.201 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-generic (5.15.0.112.112) ... 2024-06-11T16:45:43.327 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-11T16:45:43.388 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-11T16:45:43.388 INFO:teuthology.orchestra.run.smithi012.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-112-generic 2024-06-11T16:45:58.088 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-11T16:45:58.089 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub' 2024-06-11T16:45:58.118 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-11T16:45:58.148 INFO:teuthology.orchestra.run.smithi012.stdout:Generating grub configuration file ... 2024-06-11T16:45:58.466 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-11T16:45:58.486 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-11T16:45:58.824 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-11T16:45:58.829 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-11T16:45:58.897 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-11T16:45:58.903 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-11T16:45:59.066 INFO:teuthology.orchestra.run.smithi012.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-11T16:45:59.066 INFO:teuthology.orchestra.run.smithi012.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-11T16:45:59.066 INFO:teuthology.orchestra.run.smithi012.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-11T16:45:59.101 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-06-11T16:45:59.619 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-06-11T16:45:59.639 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.112.112 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-06-11T16:45:59.640 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-06-11T16:45:59.641 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-06-11T16:45:59.641 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-06-11T16:45:59.641 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-112-generic 2024-06-11T16:45:59.641 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-11T16:45:59.641 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-11T16:45:59.641 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-11T16:45:59.641 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-06-11T16:45:59.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-06-11T16:45:59.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-11T16:46:00.090 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-11T16:46:00.090 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-11T16:46:03.583 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-11T16:46:03.665 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-11T16:46:04.864 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:46:04.883 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-11T16:46:04.945 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:46:05.152 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-11T16:46:05.153 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-11T16:46:05.337 INFO:teuthology.orchestra.run.smithi012.stdout:linux-image-generic is already the newest version (5.15.0.112.112). 2024-06-11T16:46:05.337 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-11T16:46:05.338 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-11T16:46:05.338 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-11T16:46:05.338 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-11T16:46:05.338 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-11T16:46:05.338 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-11T16:46:05.339 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-11T16:46:05.378 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 0 newly installed, 0 to remove and 324 not upgraded. 2024-06-11T16:46:05.381 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-06-11T16:46:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.112.112 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-06-11T16:46:05.398 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-06-11T16:46:05.398 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-06-11T16:46:05.446 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.TZ5KIpQyq0 2024-06-11T16:46:05.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TZ5KIpQyq0 2024-06-11T16:46:05.501 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.TZ5KIpQyq0 2024-06-11T16:46:05.648 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.TZ5KIpQyq0 is 10KB 2024-06-11T16:46:05.662 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.TZ5KIpQyq0 2024-06-11T16:46:05.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-11T16:46:05.721 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:46:05.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-11T16:46:05.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-11T16:46:05.813 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-112-generic 2024-06-11T16:46:05.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo update-grub 2024-06-11T16:46:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub' 2024-06-11T16:46:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-11T16:46:07.213 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-06-11T16:46:07.401 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-11T16:46:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-11T16:46:07.644 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-11T16:46:07.650 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-11T16:46:07.694 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-11T16:46:07.699 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-11T16:46:07.844 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-11T16:46:07.844 INFO:teuthology.orchestra.run.smithi012.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-11T16:46:07.844 INFO:teuthology.orchestra.run.smithi012.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-11T16:46:07.877 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-06-11T16:46:07.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-06-11T16:46:37.918 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-11T16:46:37.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:46:37.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T16:46:56.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-06-11T16:47:05.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:47:05.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T16:48:05.351 DEBUG:teuthology.orchestra.remote:timed out 2024-06-11T16:48:17.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:48:17.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T16:48:17.624 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-06-11T16:48:18.148 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:48:18.148 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-112-generic"... 2024-06-11T16:48:18.148 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-06-11T16:48:18.193 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-112-generic 2024-06-11T16:48:18.193 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-112-generic vs 5.15.0-112-generic 2024-06-11T16:48:18.193 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-11T16:48:18.194 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-11T16:48:19.195 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-11T16:48:19.195 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-11T16:48:19.268 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-06-11T16:48:19.288 DEBUG:teuthology.parallel:result is None 2024-06-11T16:48:19.289 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-11T16:48:19.300 INFO:teuthology.task.internal:Creating test directory... 2024-06-11T16:48:19.301 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-11T16:48:19.312 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-11T16:48:19.318 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-11T16:48:19.325 INFO:teuthology.task.internal:Creating archive directory... 2024-06-11T16:48:19.325 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-11T16:48:19.372 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-11T16:48:19.379 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-11T16:48:19.379 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-11T16:48:19.427 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T16:48:19.435 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T16:48:19.437 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-11T16:48:19.444 INFO:teuthology.task.internal:Configuring sudo... 2024-06-11T16:48:19.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-11T16:48:19.498 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-11T16:48:19.507 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-11T16:48:19.508 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-11T16:48:19.545 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-11T16:48:19.594 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-11T16:48:19.642 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:48:19.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-11T16:48:19.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-06-11T16:48:19.781 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-11T16:48:19.788 INFO:teuthology.task.internal:Starting timer... 2024-06-11T16:48:19.788 INFO:teuthology.run_tasks:Running task pcp... 2024-06-11T16:48:19.799 INFO:teuthology.run_tasks:Running task selinux... 2024-06-11T16:48:19.808 DEBUG:teuthology.task.selinux:Excluding smithi012: OS 'ubuntu' does not support SELinux 2024-06-11T16:48:19.808 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-11T16:48:19.808 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-06-11T16:48:19.808 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-11T16:48:19.808 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-11T16:48:19.817 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-11T16:48:19.828 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-11T16:48:19.829 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-11T16:56:18.714 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com')] 2024-06-11T16:56:18.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:56:18.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T16:56:18.804 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-06-11T16:56:18.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-11T16:56:18.887 INFO:teuthology.run_tasks:Running task clock... 2024-06-11T16:56:18.899 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-11T16:56:18.899 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-11T16:56:18.900 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T16:56:18.963 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-11T16:56:18.963 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Command line: ntpd -gq 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: ---------------------------------------------------- 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: ntp-4 is maintained by Network Time Foundation, 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: corporation. Support and training for ntp-4 are 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: available at https://www.nwtime.org/support 2024-06-11T16:56:18.964 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: ---------------------------------------------------- 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: proto: precision = 0.069 usec (-24) 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: basedate set to 2022-02-04 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: gps base set to 2022-02-06 (week 2196) 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-11T16:56:18.965 INFO:teuthology.orchestra.run.smithi012.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-11T16:56:18.966 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: restrict ::: KOD does nothing without LIMITED. 2024-06-11T16:56:18.966 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen and drop on 0 v6wildcard [::]:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen normally on 2 lo 127.0.0.1:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen normally on 3 ens1f0 172.21.15.12:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen normally on 4 lo [::1]:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25ea%3]:123 2024-06-11T16:56:18.967 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:18 ntpd[17525]: Listening on routing socket on fd #22 for interface updates 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:19 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:21 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.967 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.968 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:23.969 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:23 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:25.964 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/rawstats.20240611: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-11T16:56:25.965 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/peerstats.20240611: Permission denied 2024-06-11T16:56:25.966 INFO:teuthology.orchestra.run.smithi012.stdout:11 Jun 16:56:25 ntpd[17525]: ntpd: time slew -0.000220 s 2024-06-11T16:56:25.966 INFO:teuthology.orchestra.run.smithi012.stdout:ntpd: time slew -0.000220s 2024-06-11T16:56:25.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-11T16:56:25.966 INFO:teuthology.orchestra.run.smithi012.stderr:11 Jun 16:56:25 ntpd[17525]: can't open /var/log/ntpstats/loopstats.20240611: Permission denied 2024-06-11T16:56:26.015 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-06-11T16:56:26.015 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-06-11T16:56:26.016 INFO:teuthology.orchestra.run.smithi012.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-11T16:56:26.016 INFO:teuthology.orchestra.run.smithi012.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-11T16:56:26.016 INFO:teuthology.orchestra.run.smithi012.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-11T16:56:26.016 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-11T16:56:26.016 INFO:teuthology.run_tasks:Running task install... 2024-06-11T16:56:26.024 DEBUG:teuthology.task.install:project ceph 2024-06-11T16:56:26.025 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'}} 2024-06-11T16:56:26.025 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'} 2024-06-11T16:56:26.025 INFO:teuthology.task.install:Using flavor: default 2024-06-11T16:56:26.032 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-11T16:56:26.032 INFO:teuthology.task.install:extra packages: [] 2024-06-11T16:56:26.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-key list | grep Ceph 2024-06-11T16:56:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-11T16:56:26.152 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-11T16:56:26.152 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph.com (release key) 2024-06-11T16:56:26.153 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-11T16:56:26.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T16:56:26.382 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default/ 2024-06-11T16:56:26.382 INFO:teuthology.task.install.deb:Package version is 19.0.0-4216-g37bbe1d8-1jammy 2024-06-11T16:56:26.494 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:56:26.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-11T16:56:26.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-11T16:56:26.686 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-11T16:56:26.699 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-11T16:56:26.719 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-11T16:56:26.798 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy InRelease 2024-06-11T16:56:26.803 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-11T16:56:26.837 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy Release [7,653 B] 2024-06-11T16:56:26.873 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-11T16:56:26.910 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,423 B] 2024-06-11T16:56:26.953 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-06-11T16:56:27.043 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 29.2 kB in 0s (82.8 kB/s) 2024-06-11T16:56:28.036 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:56:28.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-4216-g37bbe1d8-1jammy cephadm=19.0.0-4216-g37bbe1d8-1jammy ceph-mds=19.0.0-4216-g37bbe1d8-1jammy ceph-mgr=19.0.0-4216-g37bbe1d8-1jammy ceph-common=19.0.0-4216-g37bbe1d8-1jammy ceph-fuse=19.0.0-4216-g37bbe1d8-1jammy ceph-test=19.0.0-4216-g37bbe1d8-1jammy ceph-volume=19.0.0-4216-g37bbe1d8-1jammy radosgw=19.0.0-4216-g37bbe1d8-1jammy python3-rados=19.0.0-4216-g37bbe1d8-1jammy python3-rgw=19.0.0-4216-g37bbe1d8-1jammy python3-cephfs=19.0.0-4216-g37bbe1d8-1jammy python3-rbd=19.0.0-4216-g37bbe1d8-1jammy libcephfs2=19.0.0-4216-g37bbe1d8-1jammy libcephfs-dev=19.0.0-4216-g37bbe1d8-1jammy librados2=19.0.0-4216-g37bbe1d8-1jammy librbd1=19.0.0-4216-g37bbe1d8-1jammy rbd-fuse=19.0.0-4216-g37bbe1d8-1jammy 2024-06-11T16:56:28.108 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-11T16:56:28.289 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-11T16:56:28.290 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-11T16:56:28.473 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-06-11T16:56:28.475 INFO:teuthology.orchestra.run.smithi012.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-11T16:56:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-11T16:56:28.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet zip 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-06-11T16:56:28.478 INFO:teuthology.orchestra.run.smithi012.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-06-11T16:56:28.479 INFO:teuthology.orchestra.run.smithi012.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-06-11T16:56:28.479 INFO:teuthology.orchestra.run.smithi012.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-06-11T16:56:28.479 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-06-11T16:56:28.479 INFO:teuthology.orchestra.run.smithi012.stdout: btrfs-tools 2024-06-11T16:56:28.583 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-06-11T16:56:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-11T16:56:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-06-11T16:56:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-11T16:56:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-11T16:56:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-11T16:56:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-11T16:56:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-11T16:56:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-11T16:56:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-06-11T16:56:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-06-11T16:56:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-06-11T16:56:28.649 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 118 newly installed, 0 to remove and 311 not upgraded. 2024-06-11T16:56:28.650 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 190 MB of archives. 2024-06-11T16:56:28.650 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 864 MB of additional disk space will be used. 2024-06-11T16:56:28.650 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-11T16:56:28.725 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-4216-g37bbe1d8-1jammy [3,565 kB] 2024-06-11T16:56:28.755 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-11T16:56:28.758 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-11T16:56:28.778 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-11T16:56:28.847 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-11T16:56:28.852 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-11T16:56:28.865 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-11T16:56:28.870 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-11T16:56:28.870 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-11T16:56:28.871 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-11T16:56:28.880 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-06-11T16:56:28.882 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-06-11T16:56:28.885 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-06-11T16:56:28.886 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-06-11T16:56:28.897 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-06-11T16:56:28.899 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-06-11T16:56:28.902 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-06-11T16:56:28.904 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-11T16:56:28.904 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-11T16:56:28.918 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-11T16:56:28.919 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-11T16:56:28.920 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-11T16:56:28.920 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-11T16:56:28.921 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-11T16:56:28.938 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-4216-g37bbe1d8-1jammy [3,229 kB] 2024-06-11T16:56:28.938 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-11T16:56:28.939 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-11T16:56:28.939 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-11T16:56:28.942 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-11T16:56:28.943 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-11T16:56:28.959 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-11T16:56:28.960 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-11T16:56:28.960 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-11T16:56:28.961 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-11T16:56:28.961 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-11T16:56:28.980 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-11T16:56:28.980 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-11T16:56:28.981 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-11T16:56:28.986 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-11T16:56:28.987 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-11T16:56:29.000 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-11T16:56:29.001 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-11T16:56:29.002 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-11T16:56:29.004 INFO:teuthology.orchestra.run.smithi012.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-11T16:56:29.005 INFO:teuthology.orchestra.run.smithi012.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-11T16:56:29.023 INFO:teuthology.orchestra.run.smithi012.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-11T16:56:29.023 INFO:teuthology.orchestra.run.smithi012.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-11T16:56:29.026 INFO:teuthology.orchestra.run.smithi012.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-11T16:56:29.026 INFO:teuthology.orchestra.run.smithi012.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-11T16:56:29.028 INFO:teuthology.orchestra.run.smithi012.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-11T16:56:29.035 INFO:teuthology.orchestra.run.smithi012.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-4216-g37bbe1d8-1jammy [716 kB] 2024-06-11T16:56:29.045 INFO:teuthology.orchestra.run.smithi012.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-11T16:56:29.058 INFO:teuthology.orchestra.run.smithi012.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-4216-g37bbe1d8-1jammy [357 kB] 2024-06-11T16:56:29.070 INFO:teuthology.orchestra.run.smithi012.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-4216-g37bbe1d8-1jammy [32.8 kB] 2024-06-11T16:56:29.071 INFO:teuthology.orchestra.run.smithi012.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-4216-g37bbe1d8-1jammy [180 kB] 2024-06-11T16:56:29.077 INFO:teuthology.orchestra.run.smithi012.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-4216-g37bbe1d8-1jammy [67.4 kB] 2024-06-11T16:56:29.079 INFO:teuthology.orchestra.run.smithi012.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-4216-g37bbe1d8-1jammy [334 kB] 2024-06-11T16:56:29.090 INFO:teuthology.orchestra.run.smithi012.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-4216-g37bbe1d8-1jammy [7,899 kB] 2024-06-11T16:56:29.099 INFO:teuthology.orchestra.run.smithi012.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-11T16:56:29.100 INFO:teuthology.orchestra.run.smithi012.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-11T16:56:29.102 INFO:teuthology.orchestra.run.smithi012.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-11T16:56:29.102 INFO:teuthology.orchestra.run.smithi012.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-11T16:56:29.103 INFO:teuthology.orchestra.run.smithi012.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-11T16:56:29.128 INFO:teuthology.orchestra.run.smithi012.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-11T16:56:29.131 INFO:teuthology.orchestra.run.smithi012.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-11T16:56:29.132 INFO:teuthology.orchestra.run.smithi012.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-11T16:56:29.156 INFO:teuthology.orchestra.run.smithi012.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-11T16:56:29.156 INFO:teuthology.orchestra.run.smithi012.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-11T16:56:29.166 INFO:teuthology.orchestra.run.smithi012.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-11T16:56:29.167 INFO:teuthology.orchestra.run.smithi012.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-11T16:56:29.167 INFO:teuthology.orchestra.run.smithi012.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-11T16:56:29.167 INFO:teuthology.orchestra.run.smithi012.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-11T16:56:29.168 INFO:teuthology.orchestra.run.smithi012.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-11T16:56:29.168 INFO:teuthology.orchestra.run.smithi012.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-11T16:56:29.172 INFO:teuthology.orchestra.run.smithi012.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-11T16:56:29.186 INFO:teuthology.orchestra.run.smithi012.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-11T16:56:29.188 INFO:teuthology.orchestra.run.smithi012.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-11T16:56:29.189 INFO:teuthology.orchestra.run.smithi012.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-11T16:56:29.207 INFO:teuthology.orchestra.run.smithi012.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-11T16:56:29.209 INFO:teuthology.orchestra.run.smithi012.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-06-11T16:56:29.211 INFO:teuthology.orchestra.run.smithi012.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-06-11T16:56:29.211 INFO:teuthology.orchestra.run.smithi012.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-11T16:56:29.216 INFO:teuthology.orchestra.run.smithi012.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-06-11T16:56:29.216 INFO:teuthology.orchestra.run.smithi012.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-11T16:56:29.220 INFO:teuthology.orchestra.run.smithi012.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-11T16:56:29.227 INFO:teuthology.orchestra.run.smithi012.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-11T16:56:29.228 INFO:teuthology.orchestra.run.smithi012.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-11T16:56:29.229 INFO:teuthology.orchestra.run.smithi012.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-11T16:56:29.248 INFO:teuthology.orchestra.run.smithi012.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-11T16:56:29.249 INFO:teuthology.orchestra.run.smithi012.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-11T16:56:29.256 INFO:teuthology.orchestra.run.smithi012.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-11T16:56:29.257 INFO:teuthology.orchestra.run.smithi012.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-11T16:56:29.257 INFO:teuthology.orchestra.run.smithi012.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-11T16:56:29.259 INFO:teuthology.orchestra.run.smithi012.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-11T16:56:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-11T16:56:29.351 INFO:teuthology.orchestra.run.smithi012.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-4216-g37bbe1d8-1jammy [112 kB] 2024-06-11T16:56:29.354 INFO:teuthology.orchestra.run.smithi012.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-4216-g37bbe1d8-1jammy [470 kB] 2024-06-11T16:56:29.368 INFO:teuthology.orchestra.run.smithi012.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-4216-g37bbe1d8-1jammy [27.9 MB] 2024-06-11T16:56:30.401 INFO:teuthology.orchestra.run.smithi012.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-4216-g37bbe1d8-1jammy [5,095 kB] 2024-06-11T16:56:30.480 INFO:teuthology.orchestra.run.smithi012.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-4216-g37bbe1d8-1jammy [247 kB] 2024-06-11T16:56:30.483 INFO:teuthology.orchestra.run.smithi012.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-4216-g37bbe1d8-1jammy [124 kB] 2024-06-11T16:56:30.484 INFO:teuthology.orchestra.run.smithi012.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-4216-g37bbe1d8-1jammy [1,546 kB] 2024-06-11T16:56:30.527 INFO:teuthology.orchestra.run.smithi012.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-4216-g37bbe1d8-1jammy [6,265 kB] 2024-06-11T16:56:30.734 INFO:teuthology.orchestra.run.smithi012.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-4216-g37bbe1d8-1jammy [22.8 MB] 2024-06-11T16:56:31.584 INFO:teuthology.orchestra.run.smithi012.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-4216-g37bbe1d8-1jammy [14.0 kB] 2024-06-11T16:56:31.585 INFO:teuthology.orchestra.run.smithi012.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-4216-g37bbe1d8-1jammy [871 kB] 2024-06-11T16:56:31.602 INFO:teuthology.orchestra.run.smithi012.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-4216-g37bbe1d8-1jammy [2,444 kB] 2024-06-11T16:56:31.646 INFO:teuthology.orchestra.run.smithi012.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-4216-g37bbe1d8-1jammy [792 kB] 2024-06-11T16:56:31.656 INFO:teuthology.orchestra.run.smithi012.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-4216-g37bbe1d8-1jammy [149 kB] 2024-06-11T16:56:31.658 INFO:teuthology.orchestra.run.smithi012.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-4216-g37bbe1d8-1jammy [2,297 kB] 2024-06-11T16:56:31.697 INFO:teuthology.orchestra.run.smithi012.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-4216-g37bbe1d8-1jammy [8,625 kB] 2024-06-11T16:56:31.982 INFO:teuthology.orchestra.run.smithi012.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-4216-g37bbe1d8-1jammy [14.2 kB] 2024-06-11T16:56:31.982 INFO:teuthology.orchestra.run.smithi012.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-4216-g37bbe1d8-1jammy [54.8 MB] 2024-06-11T16:56:34.044 INFO:teuthology.orchestra.run.smithi012.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-4216-g37bbe1d8-1jammy [124 kB] 2024-06-11T16:56:34.046 INFO:teuthology.orchestra.run.smithi012.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-4216-g37bbe1d8-1jammy [38.9 kB] 2024-06-11T16:56:34.047 INFO:teuthology.orchestra.run.smithi012.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-4216-g37bbe1d8-1jammy [15.7 MB] 2024-06-11T16:56:34.526 INFO:teuthology.orchestra.run.smithi012.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-4216-g37bbe1d8-1jammy [91.3 kB] 2024-06-11T16:56:35.120 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 190 MB in 6s (31.9 MB/s) 2024-06-11T16:56:35.285 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-11T16:56:35.388 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165982 files and directories currently installed.) 2024-06-11T16:56:35.393 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-11T16:56:35.475 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-11T16:56:35.812 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-11T16:56:35.839 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-11T16:56:35.878 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-11T16:56:36.149 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-11T16:56:36.175 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-11T16:56:36.217 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-11T16:56:36.702 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-11T16:56:36.729 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-11T16:56:36.777 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:56:37.164 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-11T16:56:37.190 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-11T16:56:37.222 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:56:37.667 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-11T16:56:37.694 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-11T16:56:37.733 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:56:38.070 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-11T16:56:38.096 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-11T16:56:38.136 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-11T16:56:38.475 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librados2. 2024-06-11T16:56:38.501 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../007-librados2_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:38.541 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librados2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:38.994 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librbd1. 2024-06-11T16:56:39.021 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../008-librbd1_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:39.060 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librbd1 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:39.505 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs2. 2024-06-11T16:56:39.531 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:39.571 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:39.849 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rados. 2024-06-11T16:56:39.865 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../010-python3-rados_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:39.907 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rados (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:40.285 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-11T16:56:40.312 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:56:40.351 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-argparse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:40.604 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cephfs. 2024-06-11T16:56:40.631 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:40.670 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cephfs (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:40.965 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-11T16:56:40.992 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:56:41.032 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-common (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:41.335 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-11T16:56:41.362 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-11T16:56:41.393 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-11T16:56:41.704 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-prettytable. 2024-06-11T16:56:41.731 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-06-11T16:56:41.771 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-11T16:56:42.137 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rbd. 2024-06-11T16:56:42.164 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:42.265 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rbd (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:42.618 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-11T16:56:42.645 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-11T16:56:42.684 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-11T16:56:42.937 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-06-11T16:56:42.953 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-06-11T16:56:42.995 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-06-11T16:56:43.323 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-06-11T16:56:43.339 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-06-11T16:56:43.381 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-11T16:56:43.718 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua5.1. 2024-06-11T16:56:43.745 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-06-11T16:56:43.785 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-06-11T16:56:44.112 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-any. 2024-06-11T16:56:44.137 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-06-11T16:56:44.171 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-any (27ubuntu1) ... 2024-06-11T16:56:44.399 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package zip. 2024-06-11T16:56:44.415 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-06-11T16:56:44.457 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking zip (3.0-12build2) ... 2024-06-11T16:56:44.735 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package unzip. 2024-06-11T16:56:44.762 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-06-11T16:56:44.793 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-06-11T16:56:45.189 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package luarocks. 2024-06-11T16:56:45.216 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-06-11T16:56:45.255 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-06-11T16:56:45.638 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librgw2. 2024-06-11T16:56:45.665 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../025-librgw2_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:45.704 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librgw2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:46.298 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rgw. 2024-06-11T16:56:46.314 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:46.348 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rgw (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:46.718 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-11T16:56:46.744 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-11T16:56:46.784 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-11T16:56:47.096 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libradosstriper1. 2024-06-11T16:56:47.122 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:47.246 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libradosstriper1 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:47.858 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-common. 2024-06-11T16:56:47.885 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../029-ceph-common_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:47.942 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-common (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:49.554 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-base. 2024-06-11T16:56:49.581 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../030-ceph-base_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:56:49.630 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-base (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:56:50.192 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-11T16:56:50.219 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-11T16:56:50.258 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-11T16:56:50.578 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cheroot. 2024-06-11T16:56:50.605 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-11T16:56:50.644 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-11T16:56:50.973 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-11T16:56:50.999 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-11T16:56:51.039 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-11T16:56:51.317 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-11T16:56:51.344 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-11T16:56:51.383 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-11T16:56:51.653 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-11T16:56:51.680 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-11T16:56:51.719 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-11T16:56:52.039 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempora. 2024-06-11T16:56:52.066 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-06-11T16:56:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-11T16:56:52.409 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-portend. 2024-06-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-06-11T16:56:52.475 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-11T16:56:53.011 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-11T16:56:53.038 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-11T16:56:53.078 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-11T16:56:53.347 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-11T16:56:53.374 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-11T16:56:53.414 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-11T16:56:53.726 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-natsort. 2024-06-11T16:56:53.741 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-06-11T16:56:53.775 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-11T16:56:54.062 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-logutils. 2024-06-11T16:56:54.089 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-06-11T16:56:54.129 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-11T16:56:54.440 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-11T16:56:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-11T16:56:54.506 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-11T16:56:54.809 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-mako. 2024-06-11T16:56:54.837 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-11T16:56:54.876 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-11T16:56:55.212 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-11T16:56:55.240 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-11T16:56:55.279 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-11T16:56:55.565 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-11T16:56:55.592 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-11T16:56:55.648 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-11T16:56:55.951 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webob. 2024-06-11T16:56:55.978 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-11T16:56:56.018 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-11T16:56:56.379 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-waitress. 2024-06-11T16:56:56.406 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-11T16:56:56.449 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-11T16:56:56.774 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempita. 2024-06-11T16:56:56.801 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-11T16:56:56.840 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-11T16:56:57.110 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-paste. 2024-06-11T16:56:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-11T16:56:57.168 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-11T16:56:57.488 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-11T16:56:57.504 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-11T16:56:57.546 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-11T16:56:57.841 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-11T16:56:57.867 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-11T16:56:58.027 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-11T16:56:58.330 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webtest. 2024-06-11T16:56:58.354 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-06-11T16:56:58.385 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-11T16:56:58.688 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pecan. 2024-06-11T16:56:58.715 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-11T16:56:58.754 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-11T16:56:59.126 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-certifi. 2024-06-11T16:56:59.153 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-06-11T16:56:59.192 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-11T16:56:59.562 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-idna. 2024-06-11T16:56:59.589 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-11T16:56:59.628 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-11T16:56:59.898 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-urllib3. 2024-06-11T16:56:59.925 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-11T16:56:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-11T16:57:00.285 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests. 2024-06-11T16:57:00.312 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-11T16:57:00.351 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-11T16:57:00.654 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-11T16:57:00.681 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-11T16:57:00.720 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-11T16:57:00.999 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-11T16:57:01.015 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:01.057 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-modules-core (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:01.429 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-11T16:57:01.455 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:01.495 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:01.790 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-11T16:57:01.817 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-11T16:57:01.848 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-11T16:57:02.943 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-openssl. 2024-06-11T16:57:02.970 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-06-11T16:57:03.009 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-11T16:57:03.371 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr. 2024-06-11T16:57:03.398 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:03.807 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mon. 2024-06-11T16:57:03.834 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:03.873 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mon (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:04.569 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-osd. 2024-06-11T16:57:04.596 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:04.635 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-osd (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:05.838 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph. 2024-06-11T16:57:05.854 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../066-ceph_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:05.896 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:06.224 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-fuse. 2024-06-11T16:57:06.252 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:06.291 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-fuse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:06.602 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mds. 2024-06-11T16:57:06.629 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:06.669 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mds (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:07.072 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package cephadm. 2024-06-11T16:57:07.099 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../069-cephadm_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:07.155 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking cephadm (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:07.458 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-11T16:57:07.485 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-11T16:57:07.524 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-11T16:57:07.843 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-11T16:57:07.871 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-11T16:57:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-11T16:57:09.065 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-babel. 2024-06-11T16:57:09.093 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-11T16:57:09.123 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-11T16:57:09.426 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jinja2. 2024-06-11T16:57:09.453 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-11T16:57:09.484 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-11T16:57:09.804 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-11T16:57:09.820 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:09.853 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-cephadm (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:10.174 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-11T16:57:10.190 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-06-11T16:57:10.223 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-11T16:57:10.518 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-routes. 2024-06-11T16:57:10.545 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-11T16:57:10.584 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-11T16:57:10.888 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-11T16:57:10.915 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:10.979 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-dashboard (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:12.218 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-11T16:57:12.245 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-11T16:57:12.284 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-11T16:57:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-joblib. 2024-06-11T16:57:12.823 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-11T16:57:12.862 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-11T16:57:13.233 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-11T16:57:13.260 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-11T16:57:13.299 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-11T16:57:13.585 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn. 2024-06-11T16:57:13.613 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-11T16:57:13.652 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-11T16:57:14.393 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-11T16:57:14.421 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:14.545 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:15.533 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cachetools. 2024-06-11T16:57:15.560 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-06-11T16:57:15.599 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-11T16:57:15.901 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-11T16:57:15.929 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-11T16:57:15.968 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-11T16:57:16.305 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-11T16:57:16.332 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-11T16:57:16.371 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-11T16:57:16.666 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rsa. 2024-06-11T16:57:16.693 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-06-11T16:57:16.732 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-11T16:57:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-google-auth. 2024-06-11T16:57:17.039 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-06-11T16:57:17.069 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-11T16:57:17.389 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-11T16:57:17.417 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-11T16:57:17.456 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-11T16:57:17.709 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-websocket. 2024-06-11T16:57:17.736 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-06-11T16:57:17.767 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-11T16:57:18.062 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-11T16:57:18.090 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-11T16:57:18.128 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-11T16:57:18.778 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-11T16:57:18.806 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:18.844 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:19.322 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-11T16:57:19.349 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-11T16:57:19.388 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-11T16:57:19.708 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-11T16:57:19.736 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-11T16:57:19.774 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-11T16:57:20.035 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package jq. 2024-06-11T16:57:20.052 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-11T16:57:20.094 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-11T16:57:20.346 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package socat. 2024-06-11T16:57:20.362 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-11T16:57:20.404 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-11T16:57:20.708 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package xmlstarlet. 2024-06-11T16:57:20.736 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-11T16:57:20.766 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-11T16:57:21.128 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-test. 2024-06-11T16:57:21.145 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../097-ceph-test_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:21.186 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-test (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:23.941 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-volume. 2024-06-11T16:57:23.969 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-4216-g37bbe1d8-1jammy_all.deb ... 2024-06-11T16:57:24.007 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-volume (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:24.336 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-11T16:57:24.353 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:24.394 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs-dev (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:24.714 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-socket:amd64. 2024-06-11T16:57:24.741 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-06-11T16:57:24.780 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-11T16:57:25.077 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package lua-sec:amd64. 2024-06-11T16:57:25.105 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-06-11T16:57:25.135 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-06-11T16:57:25.505 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package nvme-cli. 2024-06-11T16:57:25.526 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-11T16:57:25.563 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-11T16:57:25.977 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package pkg-config. 2024-06-11T16:57:26.005 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-06-11T16:57:26.043 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-06-11T16:57:26.288 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-11T16:57:26.304 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-11T16:57:26.363 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-11T16:57:26.766 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-11T16:57:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-11T16:57:26.824 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-11T16:57:27.077 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastescript. 2024-06-11T16:57:27.105 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-06-11T16:57:27.143 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-11T16:57:27.463 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pluggy. 2024-06-11T16:57:27.484 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-11T16:57:27.521 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-11T16:57:27.816 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-psutil. 2024-06-11T16:57:27.844 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-11T16:57:27.882 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-11T16:57:28.227 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-py. 2024-06-11T16:57:28.256 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-06-11T16:57:28.293 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-11T16:57:28.623 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pygments. 2024-06-11T16:57:28.650 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-11T16:57:28.688 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-11T16:57:29.176 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-11T16:57:29.205 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-11T16:57:29.242 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-11T16:57:29.770 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-toml. 2024-06-11T16:57:29.799 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-06-11T16:57:29.836 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-11T16:57:30.139 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pytest. 2024-06-11T16:57:30.168 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-11T16:57:30.214 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-11T16:57:30.568 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplejson. 2024-06-11T16:57:30.596 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-11T16:57:30.634 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-11T16:57:30.870 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-11T16:57:30.883 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-11T16:57:30.920 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-11T16:57:31.616 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package radosgw. 2024-06-11T16:57:31.644 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../116-radosgw_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:31.683 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking radosgw (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:32.585 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package rbd-fuse. 2024-06-11T16:57:32.601 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-4216-g37bbe1d8-1jammy_amd64.deb ... 2024-06-11T16:57:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking rbd-fuse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:32.956 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-11T16:57:33.240 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-11T16:57:33.341 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-11T16:57:33.605 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-11T16:57:33.975 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-11T16:57:34.404 INFO:teuthology.orchestra.run.smithi012.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-11T16:57:34.501 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up cephadm (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:35.127 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user cephadm....done 2024-06-11T16:57:35.275 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-11T16:57:35.918 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-11T16:57:36.178 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-11T16:57:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-11T16:57:36.563 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-11T16:57:36.831 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-11T16:57:36.957 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-11T16:57:37.267 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-11T16:57:37.636 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-11T16:57:37.903 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-11T16:57:38.179 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-06-11T16:57:38.431 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-11T16:57:38.682 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-11T16:57:38.983 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-11T16:57:39.226 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-argparse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:39.502 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-11T16:57:39.628 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-11T16:57:39.754 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-06-11T16:57:39.904 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-11T16:57:40.414 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-06-11T16:57:40.782 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-06-11T16:57:40.813 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-06-11T16:57:40.876 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-11T16:57:40.985 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-11T16:57:41.361 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-11T16:57:41.487 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-11T16:57:41.746 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-11T16:57:42.018 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-11T16:57:42.324 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up zip (3.0-12build2) ... 2024-06-11T16:57:42.450 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-11T16:57:43.168 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-11T16:57:43.444 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-11T16:57:43.570 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-11T16:57:43.679 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-11T16:57:43.981 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-11T16:57:44.374 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-11T16:57:44.600 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-11T16:57:44.984 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-11T16:57:45.294 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-11T16:57:45.662 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-11T16:57:45.905 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-11T16:57:46.014 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-common (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-11T16:57:46.634 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-11T16:57:47.856 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-06-11T16:57:48.082 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:57:48.192 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-11T16:57:48.451 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-11T16:57:48.602 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-11T16:57:48.728 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-11T16:57:49.029 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-11T16:57:49.331 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-11T16:57:49.615 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:57:49.716 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-11T16:57:50.009 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-11T16:57:50.268 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-11T16:57:50.536 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-11T16:57:50.804 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-11T16:57:51.055 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-11T16:57:51.323 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-11T16:57:51.424 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-11T16:57:51.700 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-11T16:57:51.826 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-11T16:57:52.127 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-11T16:57:52.428 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-11T16:57:52.679 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-11T16:57:52.788 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-06-11T16:57:52.906 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-11T16:57:52.995 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-11T16:57:53.257 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-11T16:57:53.384 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-11T16:57:53.785 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-11T16:57:54.044 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up lua-any (27ubuntu1) ... 2024-06-11T16:57:54.145 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-11T16:57:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-11T16:57:54.514 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-11T16:57:54.865 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-11T16:57:55.175 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-11T16:57:55.476 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-11T16:57:55.585 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-11T16:57:55.878 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-11T16:57:56.146 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-11T16:57:56.499 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-11T16:57:56.876 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-11T16:57:57.169 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-11T16:57:57.538 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-11T16:57:57.664 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librados2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:57.790 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libsqlite3-mod-ceph (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:57.916 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-06-11T16:57:58.253 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:58.362 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libradosstriper1 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:58.463 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-11T16:57:58.723 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librbd1 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:58.849 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-modules-core (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:58.950 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-fuse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:59.153 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-11T16:57:59.154 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-11T16:57:59.676 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs-dev (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:59.802 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rados (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:57:59.920 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librgw2 (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:00.020 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-11T16:58:01.342 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rbd (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:01.459 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up rbd-fuse (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:01.585 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rgw (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:01.695 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cephfs (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:01.804 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-common (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:02.339 INFO:teuthology.orchestra.run.smithi012.stdout:Adding group ceph....done 2024-06-11T16:58:02.731 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user ceph....done 2024-06-11T16:58:02.909 INFO:teuthology.orchestra.run.smithi012.stdout:Setting system user ceph properties....done 2024-06-11T16:58:02.930 INFO:teuthology.orchestra.run.smithi012.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-11T16:58:03.071 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-11T16:58:03.376 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-11T16:58:03.847 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-test (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:03.973 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up radosgw (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:04.421 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-11T16:58:04.421 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-11T16:58:04.891 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-base (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:05.246 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-11T16:58:05.751 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mds (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:05.994 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-11T16:58:05.994 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-11T16:58:06.469 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:06.705 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-11T16:58:06.705 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-11T16:58:07.154 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-osd (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:07.426 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-11T16:58:07.427 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-11T16:58:07.881 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-k8sevents (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:07.982 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:08.116 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mon (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:08.329 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-11T16:58:08.330 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-11T16:58:08.801 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-cephadm (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:08.952 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:09.077 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-dashboard (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:09.237 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-volume (19.0.0-4216-g37bbe1d8-1jammy) ... 2024-06-11T16:58:09.647 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-06-11T16:58:09.816 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-11T16:58:09.952 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-11T16:58:11.419 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-06-11T16:58:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-11T16:58:12.089 DEBUG:teuthology.parallel:result is None 2024-06-11T16:58:12.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T16:58:12.233 DEBUG:teuthology.orchestra.run.smithi012:> dpkg-query -W -f '${Version}' ceph 2024-06-11T16:58:12.252 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-4216-g37bbe1d8-1jammy 2024-06-11T16:58:12.253 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4216-g37bbe1d8-1jammy 2024-06-11T16:58:12.253 INFO:teuthology.task.install:The correct ceph version 19.0.0-4216-g37bbe1d8-1jammy is installed. 2024-06-11T16:58:12.257 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-11T16:58:12.257 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:58:12.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-11T16:58:12.308 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-11T16:58:12.309 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:58:12.309 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-06-11T16:58:12.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-11T16:58:12.415 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-11T16:58:12.416 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:58:12.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-11T16:58:12.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-11T16:58:12.527 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-11T16:58:12.527 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:58:12.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-06-11T16:58:12.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-11T16:58:12.640 INFO:teuthology.run_tasks:Running task workunit... 2024-06-11T16:58:12.653 INFO:tasks.workunit:Pulling workunits from ref 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T16:58:12.654 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-11T16:58:12.654 INFO:tasks.workunit:timeout=3h 2024-06-11T16:58:12.654 INFO:tasks.workunit:cleanup=True 2024-06-11T16:58:12.654 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-11T16:58:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T16:58:12.689 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-11T16:58:12.689 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-11T16:58:12.736 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-11T16:58:12.736 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-11T16:58:12.785 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T16:58:12.835 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-11T16:59:03.665 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 97% (11996/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-06-11T16:59:04.492 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'. 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.493 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-06-11T16:59:04.494 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.494 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-11T16:59:04.494 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-06-11T16:59:04.494 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 37bbe1d8d78 Merge branch 'wip-osd-ec-partial-reads' of https://github.com/rzarzynski/ceph into wip-lflores-testing-2-2024-06-10-1925 2024-06-11T16:59:04.500 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-11T16:59:04.550 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-11T16:59:04.550 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-11T16:59:04.596 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-06-11T16:59:04.597 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-06-11T16:59:04.597 DEBUG:teuthology.orchestra.run.smithi012:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-06-11T16:59:04.658 INFO:tasks.workunit.client.0.smithi012.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-11T16:59:04.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-mapper 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-06-11T16:59:04.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-06-11T16:59:04.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none ' 2024-06-11T16:59:04.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-06-11T16:59:04.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-11T16:59:04.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-06-11T16:59:04.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-11T16:59:04.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T16:59:04.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T16:59:04.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T16:59:04.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T16:59:04.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T16:59:04.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T16:59:04.671 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T16:59:04.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T16:59:04.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T16:59:04.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T16:59:04.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T16:59:04.674 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T16:59:04.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T16:59:04.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T16:59:04.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T16:59:04.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T16:59:04.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T16:59:04.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T16:59:04.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-11T16:59:04.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T16:59:04.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T16:59:04.679 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T16:59:04.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20553 2024-06-11T16:59:04.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T16:59:04.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T16:59:04.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-06-11T16:59:04.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T16:59:04.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T16:59:04.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T16:59:04.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20553 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-06-11T16:59:04.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-06-11T16:59:04.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-06-11T16:59:04.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-06-11T16:59:04.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-06-11T16:59:04.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-06-11T16:59:04.684 INFO:tasks.workunit.client.0.smithi012.stdout:Dir: td/osd-mapper 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-06-11T16:59:04.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-06-11T16:59:04.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-06-11T16:59:04.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-11T16:59:04.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-11T16:59:04.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-11T16:59:06.741 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 3b943bbc-dfff-44f0-925f-cc277b103e70 2024-06-11T16:59:07.008 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T16:59:07.137 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.135+0000 7f51b0c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:07.161 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.159+0000 7f51b0c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:07.162 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.159+0000 7f51b0c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:07.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.159+0000 7f51b0c2c7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-11T16:59:07.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.159+0000 7f51b0c2c7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-11T16:59:07.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.159+0000 7f51b0c2c7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-11T16:59:07.200 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:07.199+0000 7f51b0c2c7c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-06-11T16:59:13.868 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T16:59:14.072 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T16:59:14.109 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:14.107+0000 7f8eef2b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:14.142 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:14.139+0000 7f8eef2b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:14.143 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:14.139+0000 7f8eef2b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:14.299 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T16:59:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:14.703+0000 7f8eef2b27c0 -1 Falling back to public interface 2024-06-11T16:59:15.526 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T16:59:16.715 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T16:59:17.437 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:17.435+0000 7f8eef2b27c0 -1 osd.0 0 log_to_monitors true 2024-06-11T16:59:18.046 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T16:59:19.463 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T16:59:20.805 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T16:59:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T16:59:23.536 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T16:59:24.872 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T16:59:26.356 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T16:59:27.694 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T16:59:29.025 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T16:59:30.379 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T16:59:31.709 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T16:59:33.325 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T16:59:34.814 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T16:59:36.508 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T16:59:38.505 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T16:59:40.607 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T16:59:42.338 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T16:59:44.185 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T16:59:44.531 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/933946201,v1:127.0.0.1:6801/933946201] [v2:127.0.0.1:6802/933946201,v1:127.0.0.1:6803/933946201] exists,up 3b943bbc-dfff-44f0-925f-cc277b103e70 2024-06-11T16:59:44.536 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 468e5554-5c01-4c09-bcbf-25ac68445524 2024-06-11T16:59:44.957 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T16:59:45.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.014+0000 7f8bc46167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:45.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.046+0000 7f8bc46167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:45.051 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.046+0000 7f8bc46167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:45.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.050+0000 7f8bc46167c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-11T16:59:45.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.050+0000 7f8bc46167c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-11T16:59:45.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.050+0000 7f8bc46167c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-11T16:59:45.122 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:45.122+0000 7f8bc46167c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-06-11T16:59:52.110 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T16:59:52.514 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T16:59:52.549 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:52.546+0000 7ff16637f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:52.578+0000 7ff16637f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:52.583 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:52.578+0000 7ff16637f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T16:59:52.862 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T16:59:53.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:53.126+0000 7ff16637f7c0 -1 Falling back to public interface 2024-06-11T16:59:54.193 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T16:59:55.517 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T16:59:55.744 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T16:59:55.738+0000 7ff16637f7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T16:59:56.943 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T16:59:58.410 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T16:59:59.871 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:00:01.257 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:00:02.726 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:00:04.081 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:00:05.419 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:00:06.888 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:00:08.231 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:00:09.719 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:00:11.300 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:00:13.001 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:00:14.677 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:00:16.320 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:00:18.189 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:00:19.894 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:00:21.293 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:00:21.625 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1818997536,v1:127.0.0.1:6811/1818997536] [v2:127.0.0.1:6812/1818997536,v1:127.0.0.1:6813/1818997536] exists,up 468e5554-5c01-4c09-bcbf-25ac68445524 2024-06-11T17:00:21.632 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 e0ad2264-2c90-4ef4-89f7-b10cf4254a82 2024-06-11T17:00:22.102 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:00:22.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.162+0000 7f4aecae87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:22.196 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.194+0000 7f4aecae87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:22.197 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.198+0000 7f4aecae87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:22.198 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.198+0000 7f4aecae87c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-11T17:00:22.198 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.198+0000 7f4aecae87c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-11T17:00:22.198 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.198+0000 7f4aecae87c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-11T17:00:22.273 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:22.274+0000 7f4aecae87c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-06-11T17:00:28.752 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:00:29.168 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:00:29.205 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:29.206+0000 7f43e1cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:29.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:29.234+0000 7f43e1cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:29.234 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:29.234+0000 7f43e1cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:00:29.499 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:00:29.782 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:29.782+0000 7f43e1cef7c0 -1 Falling back to public interface 2024-06-11T17:00:30.832 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:00:32.170 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:00:32.432 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:00:32.434+0000 7f43e1cef7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:00:33.533 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:00:34.860 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:00:36.441 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:00:37.776 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:00:39.112 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:00:40.566 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:00:41.914 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:00:43.392 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:00:44.716 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:00:46.120 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:00:47.970 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:00:49.921 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:00:51.766 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:00:53.524 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:00:55.173 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:00:56.839 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:00:58.329 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:00:58.663 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3587272526,v1:127.0.0.1:6819/3587272526] [v2:127.0.0.1:6820/3587272526,v1:127.0.0.1:6821/3587272526] exists,up e0ad2264-2c90-4ef4-89f7-b10cf4254a82 2024-06-11T17:00:59.093 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'test' created 2024-06-11T17:01:01.260 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836500 2024-06-11T17:01:02.920 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 38654705674 2024-06-11T17:01:03.271 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 60129542149 2024-06-11T17:01:06.055 INFO:tasks.workunit.client.0.smithi012.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-06-11T17:01:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-06-11T17:01:06.199 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: { 2024-06-11T17:01:06.199 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.199 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.212 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: { 2024-06-11T17:01:06.213 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.213 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.226 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2: { 2024-06-11T17:01:06.226 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.226 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-06-11T17:01:06.416 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: { 2024-06-11T17:01:06.416 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.417 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.430 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: { 2024-06-11T17:01:06.431 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.431 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.444 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2: { 2024-06-11T17:01:06.444 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-11T17:01:06.444 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:06.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-06-11T17:01:06.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-06-11T17:01:06.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-06-11T17:01:06.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-06-11T17:01:06.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-06-11T17:01:06.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-11T17:01:06.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-11T17:01:06.792 INFO:tasks.workunit.client.0.smithi012.stdout:created pool test snap snap01 2024-06-11T17:01:07.014 INFO:tasks.workunit.client.0.smithi012.stdout:created pool test snap snap02 2024-06-11T17:01:07.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-06-11T17:01:07.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-11T17:01:07.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-11T17:01:07.521 INFO:tasks.workunit.client.0.smithi012.stdout:created pool test snap snap13 2024-06-11T17:01:07.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-06-11T17:01:07.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-11T17:01:07.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-11T17:01:08.078 INFO:tasks.workunit.client.0.smithi012.stdout:created pool test snap snap24 2024-06-11T17:01:08.341 INFO:tasks.workunit.client.0.smithi012.stdout:created pool test snap snap25 2024-06-11T17:01:08.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 14581 2024-06-11T17:01:08.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-06-11T17:01:08.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-11T17:01:08.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-06-11T17:01:09.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-06-11T17:01:09.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-11T17:01:09.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-11T17:01:09.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-06-11T17:01:09.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-06-11T17:01:09.371 INFO:tasks.workunit.client.0.smithi012.stdout:pgid is 1.3 (primary: osd.1) 2024-06-11T17:01:09.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-06-11T17:01:09.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-06-11T17:01:09.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-06-11T17:01:09.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-06-11T17:01:09.713 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs_brief 2024-06-11T17:01:09.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-06-11T17:01:09.729 INFO:tasks.workunit.client.0.smithi012.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-06-11T17:01:09.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-06-11T17:01:09.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-06-11T17:01:09.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T17:01:09.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T17:01:09.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:01:09.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:01:09.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:01:09.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20553/ceph-osd.1.asok 2024-06-11T17:01:09.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-06-11T17:01:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.20553/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-06-11T17:01:09.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:09.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout:{"success":true}{ 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "objxxx", 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "clones": [ 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 2, 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 1, 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap01" 2024-06-11T17:01:09.909 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 2, 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap02" 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 3, 2024-06-11T17:01:09.910 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 3, 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap13" 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 5, 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 4, 2024-06-11T17:01:09.911 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap24" 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 5, 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap25" 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "id": "head", 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [], 2024-06-11T17:01:09.912 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6 2024-06-11T17:01:09.913 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:09.913 INFO:tasks.workunit.client.0.smithi012.stdout: ] 2024-06-11T17:01:09.913 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:09.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-11T17:01:10.039 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:10.039 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:01:10.039 INFO:tasks.workunit.client.0.smithi012.stdout: "must": true, 2024-06-11T17:01:10.039 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "0.000000" 2024-06-11T17:01:10.039 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:10.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-06-11T17:01:13.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-11T17:01:13.361 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:01:13.361 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-11T17:01:10.785450+0000 26'7 26:45 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-11T17:01:10.785248+0000 26'7 2024-06-11T17:01:10.785248+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:01:10.785248+0000 4 0 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.312381+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.350940+0000 0'0 24:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.102312+0000 0'0 26:31 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:01:13.362 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:01:13.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-06-11T17:01:13.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-11T17:01:10.785450+0000 26'7 26:45 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-11T17:01:10.785248+0000 26'7 2024-06-11T17:01:10.785248+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:01:10.785248+0000 4 0 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.312381+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.350940+0000 0'0 24:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:00.102312+0000 0'0 26:31 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:01:13.681 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:01:13.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-06-11T17:01:14.154 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:01:14.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-06-11T17:01:14.714 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:01:14.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-06-11T17:01:19.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-06-11T17:01:19.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-11T17:01:19.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:01:19.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:01:19.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:01:19.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:01:19.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:01:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:01:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-11T17:01:20.080 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-11T17:01:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.082+0000 7ff65a9d2ac0 1 bluefs mount 2024-06-11T17:01:20.094 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.094+0000 7ff65a9d2ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:20.128 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 1 bluefs mount shared_bdev_used = 0 2024-06-11T17:01:20.128 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.130+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.134+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.134+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.134+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.134+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:20.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.134+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:20.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:20.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-11T17:01:20.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-11T17:01:20.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-11T17:01:20.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-11T17:01:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 freelist init 2024-06-11T17:01:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 freelist _read_cfg 2024-06-11T17:01:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-06-11T17:01:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-06-11T17:01:20.187 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bluefs umount 2024-06-11T17:01:20.187 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.186+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) close 2024-06-11T17:01:20.360 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.362+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:20.360 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.362+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:20.360 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.362+0000 7ff65a9d2ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-11T17:01:20.360 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.362+0000 7ff65a9d2ac0 1 bluefs mount 2024-06-11T17:01:20.361 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.362+0000 7ff65a9d2ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:20.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.366+0000 7ff65a9d2ac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-11T17:01:20.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.366+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:20.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:20.371 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:20.371 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.370+0000 7ff65a9d2ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:20.412 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.410+0000 7ff65a9d2ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 0b 05 58 02 00 00 80 f1 9d 25 51 01 44 69 a4 c1 |..X......%Q.Di..| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 20 ba b9 19 1d d9 00 00 00 00 1c 00 00 00 00 00 | ...............| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:* 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-11T17:01:20.413 INFO:tasks.workunit.client.0.smithi012.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-11T17:01:20.414 INFO:tasks.workunit.client.0.smithi012.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-11T17:01:20.415 INFO:tasks.workunit.client.0.smithi012.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 3b 94 3b bc df ff |re........;.;...| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000230 44 f0 92 5f cc 27 7b 10 3e 70 00 00 00 00 00 00 |D.._.'{.>p......| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000240 00 00 1c 00 00 00 7f 82 68 66 d9 2e 13 25 01 00 |........hf...%..| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:0000025e 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:20.416 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout:0000003f 2024-06-11T17:01:20.417 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:20.418 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.419 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:20.420 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-11T17:01:20.421 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:20.422 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.414+0000 7ff65a9d2ac0 1 bluefs umount 2024-06-11T17:01:20.422 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.414+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732700 td/osd-mapper/0/block) close 2024-06-11T17:01:20.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.650+0000 7ff65a9d2ac0 1 freelist shutdown 2024-06-11T17:01:20.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.650+0000 7ff65a9d2ac0 1 bdev(0x55a2a1732000 td/osd-mapper/0/block) close 2024-06-11T17:01:20.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:20.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-11T17:01:20.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bdev(0x564c0db3c000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bdev(0x564c0db3c000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-11T17:01:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.906+0000 7f649c8c9ac0 1 bluefs mount 2024-06-11T17:01:20.922 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.922+0000 7f649c8c9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:20.957 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.958+0000 7f649c8c9ac0 1 bluefs mount shared_bdev_used = 0 2024-06-11T17:01:20.957 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.958+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:20.960 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.958+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:20.960 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:20.960 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:20.962+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:21.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:21.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-11T17:01:21.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-11T17:01:21.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 freelist init 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.014+0000 7f649c8c9ac0 1 freelist _read_cfg 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.018+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.018+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2024-06-11T17:01:21.017 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.018+0000 7f649c8c9ac0 1 bluefs umount 2024-06-11T17:01:21.018 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.018+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) close 2024-06-11T17:01:21.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.190+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:21.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.190+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:21.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.190+0000 7f649c8c9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-11T17:01:21.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.190+0000 7f649c8c9ac0 1 bluefs mount 2024-06-11T17:01:21.197 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.198+0000 7f649c8c9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:21.206 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.206+0000 7f649c8c9ac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-11T17:01:21.206 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.206+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:21.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:21.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:21.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:21.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:21.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:21.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.210+0000 7f649c8c9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:21.252 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.254+0000 7f649c8c9ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:21.253 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.254+0000 7f649c8c9ac0 1 bluefs umount 2024-06-11T17:01:21.253 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.254+0000 7f649c8c9ac0 1 bdev(0x564c0db3c700 td/osd-mapper/2/block) close 2024-06-11T17:01:21.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.478+0000 7f649c8c9ac0 1 freelist shutdown 2024-06-11T17:01:21.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:21.478+0000 7f649c8c9ac0 1 bdev(0x564c0db3c000 td/osd-mapper/2/block) close 2024-06-11T17:01:21.690 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:21.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-06-11T17:01:21.692 INFO:tasks.workunit.client.0.smithi012.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-06-11T17:01:21.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:21.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-11T17:01:22.128 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 0b 05 58 02 00 00 80 f1 9d 25 51 01 44 69 a4 c1 |..X......%Q.Di..| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 20 ba b9 19 1d d9 02 00 00 00 1c 00 00 00 00 00 | ...............| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:* 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-11T17:01:22.129 INFO:tasks.workunit.client.0.smithi012.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-11T17:01:22.130 INFO:tasks.workunit.client.0.smithi012.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000220 72 65 1c 00 00 00 0e 00 00 00 e0 ad 22 64 2c 90 |re.........."d,.| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000230 4e f4 89 f7 b1 0c f4 25 4a 82 00 00 00 00 00 00 |N......%J.......| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000240 00 00 1c 00 00 00 c9 82 68 66 b6 c7 40 04 01 00 |........hf..@...| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-11T17:01:22.131 INFO:tasks.workunit.client.0.smithi012.stdout:0000025e 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:22.132 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:0000003f 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:22.133 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:22.134 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:22.135 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-11T17:01:22.136 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:22.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-11T17:01:22.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-06-11T17:01:22.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:01:22.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-06-11T17:01:23.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-06-11T17:01:23.332 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-06-11T17:01:23.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-06-11T17:01:23.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:23.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-06-11T17:01:23.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-06-11T17:01:23.335 INFO:tasks.workunit.client.0.smithi012.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-06-11T17:01:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-11T17:01:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-11T17:01:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bdev(0x560f42954000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bdev(0x560f42954000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-11T17:01:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bluefs mount 2024-06-11T17:01:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.366+0000 7fa396dccac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:23.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.374+0000 7fa396dccac0 1 bluefs mount shared_bdev_used = 0 2024-06-11T17:01:23.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.378+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:23.426 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:23.426 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-11T17:01:23.426 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-11T17:01:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-11T17:01:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-11T17:01:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 freelist init 2024-06-11T17:01:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 freelist _read_cfg 2024-06-11T17:01:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-06-11T17:01:23.428 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-06-11T17:01:23.428 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bluefs umount 2024-06-11T17:01:23.428 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.426+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) close 2024-06-11T17:01:23.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.650+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-11T17:01:23.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.650+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:23.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.650+0000 7fa396dccac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-11T17:01:23.649 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.650+0000 7fa396dccac0 1 bluefs mount 2024-06-11T17:01:23.651 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.650+0000 7fa396dccac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:23.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.662+0000 7fa396dccac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-11T17:01:23.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.662+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.666+0000 7fa396dccac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:23.723 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.722+0000 7fa396dccac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 0b 05 58 02 00 00 80 f1 9d 25 51 01 44 69 a4 c1 |..X......%Q.Di..| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 20 ba b9 19 1d d9 00 00 00 00 1c 00 00 00 00 00 | ...............| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:* 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-11T17:01:23.725 INFO:tasks.workunit.client.0.smithi012.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-11T17:01:23.726 INFO:tasks.workunit.client.0.smithi012.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 3b 94 3b bc df ff |re........;.;...| 2024-06-11T17:01:23.727 INFO:tasks.workunit.client.0.smithi012.stdout:00000230 44 f0 92 5f cc 27 7b 10 3e 70 00 00 00 00 00 00 |D.._.'{.>p......| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000240 00 00 1c 00 00 00 7f 82 68 66 d9 2e 13 25 01 00 |........hf...%..| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:0000025e 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:23.728 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:0000003f 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:23.729 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:23.733 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.722+0000 7fa396dccac0 1 bluefs umount 2024-06-11T17:01:23.734 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.722+0000 7fa396dccac0 1 bdev(0x560f42954700 td/osd-mapper/0/block) close 2024-06-11T17:01:23.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.930+0000 7fa396dccac0 1 freelist shutdown 2024-06-11T17:01:23.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:23.930+0000 7fa396dccac0 1 bdev(0x560f42954000 td/osd-mapper/0/block) close 2024-06-11T17:01:24.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-11T17:01:24.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-11T17:01:24.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:01:24.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-11T17:01:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-11T17:01:24.985 INFO:tasks.workunit.client.0.smithi012.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:24.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-11T17:01:24.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.KyQUH7salJ_the_val 2024-06-11T17:01:24.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:24.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.KyQUH7salJ_the_val' 2024-06-11T17:01:24.988 INFO:tasks.workunit.client.0.smithi012.stdout:Value dumped in: /tmp/tmp.KyQUH7salJ_the_val 2024-06-11T17:01:24.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.KyQUH7salJ_the_val 2024-06-11T17:01:25.370 INFO:tasks.workunit.client.0.smithi012.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-11T17:01:25.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:25.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.KyQUH7salJ_the_val 2024-06-11T17:01:25.823 INFO:tasks.workunit.client.0.smithi012.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-11T17:01:25.823 INFO:tasks.workunit.client.0.smithi012.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-11T17:01:25.823 INFO:tasks.workunit.client.0.smithi012.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-11T17:01:25.823 INFO:tasks.workunit.client.0.smithi012.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stdout:0000073 2024-06-11T17:01:25.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-11T17:01:25.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:27.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.KyQUH7salJ_the_val 2024-06-11T17:01:28.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:28.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:28.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-11T17:01:28.978 INFO:tasks.workunit.client.0.smithi012.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-06-11T17:01:28.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-06-11T17:01:28.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-11T17:01:28.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-11T17:01:29.009 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bdev(0x559fef730000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-11T17:01:29.009 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bdev(0x559fef730000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:29.009 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-11T17:01:29.009 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-11T17:01:29.010 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:29.010 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-11T17:01:29.010 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.010+0000 7fb6dcd0dac0 1 bluefs mount 2024-06-11T17:01:29.015 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.014+0000 7fb6dcd0dac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:29.024 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 1 bluefs mount shared_bdev_used = 0 2024-06-11T17:01:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.026+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:29.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.030+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:29.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.030+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:29.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.030+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:29.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.030+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:29.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.030+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:29.076 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.074+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.074+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.074+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.074+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.074+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 freelist init 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 freelist _read_cfg 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-06-11T17:01:29.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 6.10353e-07 2024-06-11T17:01:29.078 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 bluefs umount 2024-06-11T17:01:29.078 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.078+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) close 2024-06-11T17:01:29.288 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.290+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-11T17:01:29.289 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.290+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:29.289 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.290+0000 7fb6dcd0dac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-11T17:01:29.289 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.290+0000 7fb6dcd0dac0 1 bluefs mount 2024-06-11T17:01:29.289 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.290+0000 7fb6dcd0dac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:29.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-11T17:01:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:29.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.298+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.302+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.302+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.302+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.302+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:29.301 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.302+0000 7fb6dcd0dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:29.356 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.358+0000 7fb6dcd0dac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:29.357 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 0b 05 58 02 00 00 80 f1 9d 25 51 01 44 69 a4 c1 |..X......%Q.Di..| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 20 ba b9 19 1d d9 01 00 00 00 1c 00 00 00 00 00 | ...............| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:* 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-11T17:01:29.358 INFO:tasks.workunit.client.0.smithi012.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-11T17:01:29.359 INFO:tasks.workunit.client.0.smithi012.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000220 72 65 1c 00 00 00 09 00 00 00 46 8e 55 54 5c 01 |re........F.UT\.| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000230 4c 09 bc bf 25 ac 68 44 55 24 00 00 00 00 00 00 |L...%.hDU$......| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000240 00 00 1c 00 00 00 a4 82 68 66 7d 75 84 09 01 00 |........hf}u....| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout:0000025e 2024-06-11T17:01:29.360 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-11T17:01:29.361 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:0000003f 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:29.362 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:29.363 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:29.364 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-11T17:01:29.365 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.366 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-11T17:01:29.366 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:29.366 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.358+0000 7fb6dcd0dac0 1 bluefs umount 2024-06-11T17:01:29.366 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.358+0000 7fb6dcd0dac0 1 bdev(0x559fef730700 td/osd-mapper/1/block) close 2024-06-11T17:01:29.568 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.570+0000 7fb6dcd0dac0 1 freelist shutdown 2024-06-11T17:01:29.568 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:29.570+0000 7fb6dcd0dac0 1 bdev(0x559fef730000 td/osd-mapper/1/block) close 2024-06-11T17:01:29.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-11T17:01:29.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-11T17:01:29.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:01:30.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:30.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:30.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-11T17:01:30.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-11T17:01:30.618 INFO:tasks.workunit.client.0.smithi012.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:30.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-11T17:01:30.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.iL5jE0rfuz_the_val 2024-06-11T17:01:30.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:30.620 INFO:tasks.workunit.client.0.smithi012.stdout:Value dumped in: /tmp/tmp.iL5jE0rfuz_the_val 2024-06-11T17:01:30.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.iL5jE0rfuz_the_val' 2024-06-11T17:01:30.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.iL5jE0rfuz_the_val 2024-06-11T17:01:31.003 INFO:tasks.workunit.client.0.smithi012.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-11T17:01:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.iL5jE0rfuz_the_val 2024-06-11T17:01:31.443 INFO:tasks.workunit.client.0.smithi012.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-11T17:01:31.443 INFO:tasks.workunit.client.0.smithi012.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-11T17:01:31.443 INFO:tasks.workunit.client.0.smithi012.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stdout:0000073 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-11T17:01:31.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:33.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.iL5jE0rfuz_the_val 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-06-11T17:01:34.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-11T17:01:34.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.374+0000 7fdea008fac0 1 bdev(0x5623fa62a000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.374+0000 7fdea008fac0 1 bdev(0x5623fa62a000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.378+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.378+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.378+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.378+0000 7fdea008fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-11T17:01:34.377 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.378+0000 7fdea008fac0 1 bluefs mount 2024-06-11T17:01:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.386+0000 7fdea008fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:34.395 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.394+0000 7fdea008fac0 1 bluefs mount shared_bdev_used = 0 2024-06-11T17:01:34.395 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.394+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:34.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:34.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:34.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:34.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:34.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:34.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.398+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.446+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.446+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.446+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.446+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.446+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 freelist init 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 freelist _read_cfg 2024-06-11T17:01:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-06-11T17:01:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2024-06-11T17:01:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 bluefs umount 2024-06-11T17:01:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.450+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) close 2024-06-11T17:01:34.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.658+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-11T17:01:34.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.658+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-11T17:01:34.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.658+0000 7fdea008fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-11T17:01:34.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.658+0000 7fdea008fac0 1 bluefs mount 2024-06-11T17:01:34.657 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.658+0000 7fdea008fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-11T17:01:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.662+0000 7fdea008fac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-11T17:01:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.662+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-11T17:01:34.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-11T17:01:34.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-11T17:01:34.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-11T17:01:34.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-11T17:01:34.665 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-11T17:01:34.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.666+0000 7fdea008fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-11T17:01:34.705 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.706+0000 7fdea008fac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-11T17:01:34.707 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-11T17:01:34.707 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 0b 05 58 02 00 00 80 f1 9d 25 51 01 44 69 a4 c1 |..X......%Q.Di..| 2024-06-11T17:01:34.707 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 20 ba b9 19 1d d9 02 00 00 00 1c 00 00 00 00 00 | ...............| 2024-06-11T17:01:34.707 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-06-11T17:01:34.707 INFO:tasks.workunit.client.0.smithi012.stdout:* 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-06-11T17:01:34.708 INFO:tasks.workunit.client.0.smithi012.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-06-11T17:01:34.709 INFO:tasks.workunit.client.0.smithi012.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000220 72 65 1c 00 00 00 0e 00 00 00 e0 ad 22 64 2c 90 |re.........."d,.| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000230 4e f4 89 f7 b1 0c f4 25 4a 82 00 00 00 00 00 00 |N......%J.......| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000240 00 00 1c 00 00 00 c9 82 68 66 b6 c7 40 04 01 00 |........hf..@...| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:0000025e 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:34.710 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:0000003f 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:34.711 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000047 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:34.712 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:34.713 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-06-11T17:01:34.714 INFO:tasks.workunit.client.0.smithi012.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout:0000003b 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.706+0000 7fdea008fac0 1 bluefs umount 2024-06-11T17:01:34.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.706+0000 7fdea008fac0 1 bdev(0x5623fa62a700 td/osd-mapper/2/block) close 2024-06-11T17:01:34.940 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.942+0000 7fdea008fac0 1 freelist shutdown 2024-06-11T17:01:34.940 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:34.942+0000 7fdea008fac0 1 bdev(0x5623fa62a000 td/osd-mapper/2/block) close 2024-06-11T17:01:35.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-11T17:01:35.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-06-11T17:01:35.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:01:35.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:35.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:35.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-11T17:01:35.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-11T17:01:35.985 INFO:tasks.workunit.client.0.smithi012.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:35.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-11T17:01:35.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.CNQilxPvrm_the_val 2024-06-11T17:01:35.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:35.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.CNQilxPvrm_the_val' 2024-06-11T17:01:35.987 INFO:tasks.workunit.client.0.smithi012.stdout:Value dumped in: /tmp/tmp.CNQilxPvrm_the_val 2024-06-11T17:01:35.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.CNQilxPvrm_the_val 2024-06-11T17:01:36.379 INFO:tasks.workunit.client.0.smithi012.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-11T17:01:36.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:36.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.CNQilxPvrm_the_val 2024-06-11T17:01:36.816 INFO:tasks.workunit.client.0.smithi012.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-11T17:01:36.816 INFO:tasks.workunit.client.0.smithi012.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-11T17:01:36.816 INFO:tasks.workunit.client.0.smithi012.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-11T17:01:36.816 INFO:tasks.workunit.client.0.smithi012.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stdout:0000073 2024-06-11T17:01:36.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-11T17:01:36.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:01:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.CNQilxPvrm_the_val 2024-06-11T17:01:39.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:01:39.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-06-11T17:01:39.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-06-11T17:01:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:01:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-06-11T17:01:39.971 INFO:tasks.workunit.client.0.smithi012.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-06-11T17:01:39.972 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-06-11T17:01:39.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-06-11T17:01:39.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:39.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:39.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-06-11T17:01:39.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:01:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:01:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:01:39.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-06-11T17:01:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:01:39.982 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:01:39.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:01:39.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-06-11T17:01:39.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:01:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:01:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:01:39.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:01:40.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:40.030+0000 7f7fc5cde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:40.049 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:40.050+0000 7f7fc5cde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:40.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:40.050+0000 7f7fc5cde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:01:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:40.336 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:01:40.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:01:40.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:40.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:01:40.594 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:40.594+0000 7f7fc5cde7c0 -1 Falling back to public interface 2024-06-11T17:01:40.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:41.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:41.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:41.696 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:01:41.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:01:41.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:41.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:01:42.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:43.053 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:01:43.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:01:43.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:43.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:01:43.175 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:43.174+0000 7f7fc5cde7c0 -1 osd.0 28 log_to_monitors true 2024-06-11T17:01:43.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:44.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:44.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:44.409 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:01:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:01:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:01:44.757 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6800/61226621,v1:127.0.0.1:6801/61226621] [v2:127.0.0.1:6802/61226621,v1:127.0.0.1:6803/61226621] exists,up 3b943bbc-dfff-44f0-925f-cc277b103e70 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-11T17:01:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-06-11T17:01:44.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:01:44.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:44.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:44.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:01:44.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-11T17:01:44.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-06-11T17:01:44.764 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:01:44.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:01:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:01:44.813 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:44.814+0000 7ff146a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:44.818+0000 7ff146a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:44.820 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:44.822+0000 7ff146a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:45.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:01:45.104 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:01:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:01:45.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:45.362+0000 7ff146a907c0 -1 Falling back to public interface 2024-06-11T17:01:45.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:46.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:46.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:46.440 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:01:46.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:01:46.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:01:46.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:47.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:47.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:47.783 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:01:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:01:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:01:47.886 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:47.885+0000 7ff146a907c0 -1 osd.1 28 log_to_monitors true 2024-06-11T17:01:48.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:49.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:49.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:49.124 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:01:49.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:01:49.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:49.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:01:49.577 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 29 last_clean_interval [9,28) [v2:127.0.0.1:6810/543270961,v1:127.0.0.1:6811/543270961] [v2:127.0.0.1:6812/543270961,v1:127.0.0.1:6813/543270961] exists,up 468e5554-5c01-4c09-bcbf-25ac68445524 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:01:49.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:01:49.581 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:01:49.582 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:01:49.582 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:01:49.582 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:01:49.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:01:49.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:01:49.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:01:49.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-06-11T17:01:49.584 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=80f19d25-5101-4469-a4c1-20bab9191dd9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20553/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:01:49.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:01:49.628 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:49.629+0000 7f79750ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:49.681 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:49.681+0000 7f79750ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:49.683 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:49.681+0000 7f79750ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:01:49.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T17:01:49.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:01:49.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:01:49.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:01:49.923 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:01:49.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:01:49.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:49.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:01:49.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:49.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:01:50.230 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:50.229+0000 7f79750ef7c0 -1 Falling back to public interface 2024-06-11T17:01:50.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:51.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:51.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:51.306 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:01:51.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:01:51.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:51.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:01:51.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:52.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:52.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:52.657 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:01:52.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:01:52.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:52.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:01:52.982 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:01:52.981+0000 7f79750ef7c0 -1 osd.2 28 log_to_monitors true 2024-06-11T17:01:53.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:01:54.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:01:54.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:01:54.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:01:54.024 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:01:54.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:01:54.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:01:54.382 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [14,28) [v2:127.0.0.1:6818/1999511862,v1:127.0.0.1:6819/1999511862] [v2:127.0.0.1:6820/1999511862,v1:127.0.0.1:6821/1999511862] exists,up e0ad2264-2c90-4ef4-89f7-b10cf4254a82 2024-06-11T17:01:54.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:01:54.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:01:54.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:01:54.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-06-11T17:01:55.385 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-06-11T17:01:55.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-06-11T17:01:55.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:55.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-06-11T17:01:55.535 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:55.535 INFO:tasks.workunit.client.0.smithi012.stdout: "version": "19.0.0-4216-g37bbe1d8", 2024-06-11T17:01:55.535 INFO:tasks.workunit.client.0.smithi012.stdout: "release": "squid", 2024-06-11T17:01:55.535 INFO:tasks.workunit.client.0.smithi012.stdout: "release_type": "dev" 2024-06-11T17:01:55.535 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:55.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:55.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-06-11T17:01:55.681 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:55.681 INFO:tasks.workunit.client.0.smithi012.stdout: "version": "19.0.0-4216-g37bbe1d8", 2024-06-11T17:01:55.681 INFO:tasks.workunit.client.0.smithi012.stdout: "release": "squid", 2024-06-11T17:01:55.681 INFO:tasks.workunit.client.0.smithi012.stdout: "release_type": "dev" 2024-06-11T17:01:55.681 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:55.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-11T17:01:55.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-06-11T17:01:55.824 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:55.824 INFO:tasks.workunit.client.0.smithi012.stdout: "version": "19.0.0-4216-g37bbe1d8", 2024-06-11T17:01:55.824 INFO:tasks.workunit.client.0.smithi012.stdout: "release": "squid", 2024-06-11T17:01:55.824 INFO:tasks.workunit.client.0.smithi012.stdout: "release_type": "dev" 2024-06-11T17:01:55.824 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:55.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "objxxx", 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "clones": [ 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 2, 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 1, 2024-06-11T17:01:55.868 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap01" 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 2, 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap02" 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 3, 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:55.869 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 3, 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap13" 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 5, 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [ 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.870 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 4, 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap24" 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "id": 5, 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "snap25" 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6, 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "overlaps": [] 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: { 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "id": "head", 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "snapshots": [], 2024-06-11T17:01:55.871 INFO:tasks.workunit.client.0.smithi012.stdout: "size": 6 2024-06-11T17:01:55.872 INFO:tasks.workunit.client.0.smithi012.stdout: } 2024-06-11T17:01:55.872 INFO:tasks.workunit.client.0.smithi012.stdout: ] 2024-06-11T17:01:55.872 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:55.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-06-11T17:01:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is unset 2024-06-11T17:01:56.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-06-11T17:01:56.792 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is unset 2024-06-11T17:01:56.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-11T17:01:56.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-11T17:01:57.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-06-11T17:01:57.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-11T17:01:57.514 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:01:57.514 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 4 0 0 24 0 0 7 0 7 active+undersized+degraded 2024-06-11T17:01:48.998434+0000 26'7 36:53 [1,0] 1 [1,0] 1 26'7 2024-06-11T17:01:10.785248+0000 26'7 2024-06-11T17:01:10.785248+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:01:10.785248+0000 4 0 2024-06-11T17:01:57.514 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-11T17:01:49.598841+0000 0'0 36:27 [0,1] 0 [0,1] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:01:57.515 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-11T17:01:49.598871+0000 0'0 36:18 [0,1] 0 [0,1] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:01:57.515 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-11T17:01:49.339802+0000 0'0 36:21 [1,0] 1 [1,0] 1 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:01:57.515 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:01:57.515 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:01:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:01:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-06-11T17:01:59.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-11T17:01:59.676 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:01:59.676 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:01:59.676 INFO:tasks.workunit.client.0.smithi012.stdout: "must": true, 2024-06-11T17:01:59.676 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "0.000000" 2024-06-11T17:01:59.676 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:01:59.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-06-11T17:02:04.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-11T17:02:05.008 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-11T17:01:59.931224+0000 26'7 42:80 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-11T17:01:59.931016+0000 26'7 2024-06-11T17:01:59.931016+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:01:59.931016+0000 4 0 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.330278+0000 0'0 42:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.384678+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.992226+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:02:05.009 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:02:05.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:02:05.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-11T17:02:05.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:01:59.925+0000 7ff129217640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-11T17:02:05.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-06-11T17:02:05.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-06-11T17:02:05.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-11T17:02:05.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-06-11T17:02:05.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-06-11T17:02:05.035 INFO:tasks.workunit.client.0.smithi012.stdout:prev count: 1 2024-06-11T17:02:05.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-11T17:02:05.177 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:02:05.177 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:02:05.177 INFO:tasks.workunit.client.0.smithi012.stdout: "must": true, 2024-06-11T17:02:05.177 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "0.000000" 2024-06-11T17:02:05.177 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:02:05.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-06-11T17:02:10.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-11T17:02:10.500 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-11T17:02:05.906040+0000 26'7 42:89 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-11T17:02:05.905685+0000 26'7 2024-06-11T17:02:05.905685+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:02:05.905685+0000 4 0 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.330278+0000 0'0 42:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.384678+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:01:54.992226+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-11T17:00:59.032246+0000 0'0 2024-06-11T17:00:59.032246+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:00:59.032246+0000 0 0 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:02:10.501 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:02:10.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:02:10.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-11T17:02:10.518 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:01:59.925+0000 7ff129217640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-11T17:02:10.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-06-11T17:02:10.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-11T17:02:10.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-06-11T17:02:10.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-11T17:02:10.522 INFO:tasks.workunit.client.0.smithi012.stdout:current count: 1 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:02:10.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:02:10.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:02:10.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:02:10.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:02:10.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-11T17:02:10.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-11T17:02:10.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-11T17:02:10.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-06-11T17:02:10.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:02:11.645 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:02:11.645 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-11T17:02:11.645 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-11T17:02:11.645 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-11T17:02:11.646 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:02:11.646 INFO:tasks.workunit.client.0.smithi012.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-11T17:02:11.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-11T17:02:11.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-06-11T17:02:11.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-11T17:02:11.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:02:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-11T17:02:12.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:02:12.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:02:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:02:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:02:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:02:12.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:02:12.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:02:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:02:12.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:02:12.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:02:12.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:02:12.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:02:12.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:02:12.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:02:12.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:02:12.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:02:12.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-11T17:02:12.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:02:12.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:12.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:02:12.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20553 2024-06-11T17:02:12.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-mapper 0 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-11T17:02:12.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-11T17:02:12.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:02:12.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:02:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:02:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:02:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:02:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:02:12.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:02:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:02:12.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:02:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:02:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:02:12.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:02:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:02:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:02:12.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:02:12.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-11T17:02:12.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20553 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20553 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:02:12.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:02:12.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-11T17:02:12.613 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-11T17:02:12.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-11T17:02:12.665 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-06-11T17:02:12.665 DEBUG:teuthology.orchestra.run.smithi012:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-06-11T17:02:12.718 INFO:tasks.workunit.client.0.smithi012.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-11T17:02:12.724 INFO:tasks.workunit.client.0.smithi012.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-11T17:02:12.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-scrub 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-06-11T17:02:12.725 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-06-11T17:02:12.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none ' 2024-06-11T17:02:12.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-11T17:02:12.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-06-11T17:02:12.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-11T17:02:12.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-06-11T17:02:12.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_recovery_scrub_2' 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-06-11T17:02:12.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.36888 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:02:12.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:02:12.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:02:12.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:02:12.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:02:12.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:02:12.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:02:12.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:02:12.735 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:02:12.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:02:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:02:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-11T17:02:12.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:02:12.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:12.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:12.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36888 2024-06-11T17:02:12.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:02:12.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:02:12.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-11T17:02:12.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:02:12.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:12.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:12.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36888 2024-06-11T17:02:12.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-11T17:02:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:02:13.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:02:13.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:13.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:13.207 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:13.207 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:13.208 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:13.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:13.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:02:13.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:02:13.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:02:13.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:02:13.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:02:13.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:02:13.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:02:13.428 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:02:13.429 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:02:13.429 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:02:13.429 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:02:13.429 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:13.429 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:13.430 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36888/ceph-mon.a.asok 2024-06-11T17:02:13.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:02:13.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36888/ceph-mon.a.asok config get fsid 2024-06-11T17:02:13.551 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:02:13.551 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:02:13.551 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:02:13.551 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:13.552 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36888/ceph-mon.a.asok 2024-06-11T17:02:13.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:02:13.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36888/ceph-mon.a.asok config get mon_host 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-11T17:02:13.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:02:14.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:02:14.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:14.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:14.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:14.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:14.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:14.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:14.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:02:14.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:02:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-11T17:02:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-11T17:02:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-11T17:02:14.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-11T17:02:14.591 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-06-11T17:02:14.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-06-11T17:02:14.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:02:14.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-06-11T17:02:14.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:02:14.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:02:14.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:02:14.595 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:02:14.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=087f645d-0461-4f37-a68b-0456b2b633d0 2024-06-11T17:02:14.597 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 087f645d-0461-4f37-a68b-0456b2b633d0 2024-06-11T17:02:14.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 087f645d-0461-4f37-a68b-0456b2b633d0' 2024-06-11T17:02:14.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:02:14.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWg2hmB9K0JBAASOewj5+T5y/JEn0+fGQR4w== 2024-06-11T17:02:14.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWg2hmB9K0JBAASOewj5+T5y/JEn0+fGQR4w=="}' 2024-06-11T17:02:14.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 087f645d-0461-4f37-a68b-0456b2b633d0 -i td/osd-recovery-scrub/0/new.json 2024-06-11T17:02:14.845 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-11T17:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAWg2hmB9K0JBAASOewj5+T5y/JEn0+fGQR4w== --osd-uuid 087f645d-0461-4f37-a68b-0456b2b633d0 2024-06-11T17:02:14.902 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.901+0000 7f0bddcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:14.905 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.905+0000 7f0bddcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.905+0000 7f0bddcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.905+0000 7f0bddcb27c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:02:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.905+0000 7f0bddcb27c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:02:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.905+0000 7f0bddcb27c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:02:14.989 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:14.989+0000 7f0bddcb27c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-11T17:02:21.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-11T17:02:21.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:02:21.528 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:02:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:02:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:02:21.945 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:02:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:02:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-11T17:02:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:02:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:02:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:02:21.979 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:21.977+0000 7f6a84a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:22.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:22.013+0000 7f6a84a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:22.013 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:22.013+0000 7f6a84a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:22.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:22.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:02:22.294 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:02:22.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:22.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:02:22.554 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:22.553+0000 7f6a84a377c0 -1 Falling back to public interface 2024-06-11T17:02:22.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:23.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:23.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:23.647 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:02:23.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:02:23.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:23.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:02:23.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:24.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:24.983 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:02:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:02:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:24.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:02:25.111 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:25.109+0000 7f6a84a377c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:02:25.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:26.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:26.325 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:02:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:02:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:26.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:02:26.675 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/811505520,v1:127.0.0.1:6803/811505520] [v2:127.0.0.1:6804/811505520,v1:127.0.0.1:6805/811505520] exists,up 087f645d-0461-4f37-a68b-0456b2b633d0 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:02:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:02:26.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:02:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4cf98280-c6bf-4fac-b08e-53a907d76696 2024-06-11T17:02:26.682 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 4cf98280-c6bf-4fac-b08e-53a907d76696 2024-06-11T17:02:26.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4cf98280-c6bf-4fac-b08e-53a907d76696' 2024-06-11T17:02:26.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:02:26.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAig2hmdTUIKhAAKfaEGXq3Y4idO6juMkIhsg== 2024-06-11T17:02:26.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAig2hmdTUIKhAAKfaEGXq3Y4idO6juMkIhsg=="}' 2024-06-11T17:02:26.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4cf98280-c6bf-4fac-b08e-53a907d76696 -i td/osd-recovery-scrub/1/new.json 2024-06-11T17:02:27.073 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:02:27.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-11T17:02:27.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAig2hmdTUIKhAAKfaEGXq3Y4idO6juMkIhsg== --osd-uuid 4cf98280-c6bf-4fac-b08e-53a907d76696 2024-06-11T17:02:27.126 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.125+0000 7f9a540437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:27.157 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.157+0000 7f9a540437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:27.159 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.157+0000 7f9a540437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:27.159 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.157+0000 7f9a540437c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:02:27.159 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.157+0000 7f9a540437c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:02:27.160 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.157+0000 7f9a540437c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:02:27.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:27.233+0000 7f9a540437c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-11T17:02:33.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-11T17:02:33.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:02:33.690 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:02:33.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:02:33.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:02:34.103 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:02:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:02:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:02:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-11T17:02:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:02:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:02:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:34.137+0000 7f955ec767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:34.171 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:34.169+0000 7f955ec767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:34.173 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:34.173+0000 7f955ec767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:34.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:02:34.436 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:02:34.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:34.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:02:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:34.721+0000 7f955ec767c0 -1 Falling back to public interface 2024-06-11T17:02:34.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:35.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:35.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:35.778 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:02:35.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:02:35.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:35.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:02:36.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:37.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:37.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:37.106 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:02:37.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:02:37.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:37.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:02:37.293 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:37.293+0000 7f955ec767c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:02:37.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:38.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:38.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:38.526 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:02:38.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:02:38.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:38.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:02:38.860 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1835112891,v1:127.0.0.1:6811/1835112891] [v2:127.0.0.1:6812/1835112891,v1:127.0.0.1:6813/1835112891] exists,up 4cf98280-c6bf-4fac-b08e-53a907d76696 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-11T17:02:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:02:38.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:02:38.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9de08238-70ac-466b-8ab4-2c16a17aa6f8 2024-06-11T17:02:38.867 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 9de08238-70ac-466b-8ab4-2c16a17aa6f8 2024-06-11T17:02:38.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9de08238-70ac-466b-8ab4-2c16a17aa6f8' 2024-06-11T17:02:38.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:02:38.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAug2hmAB8bNRAAobiNMcKY9h4AXSRY9cuk+g== 2024-06-11T17:02:38.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAug2hmAB8bNRAAobiNMcKY9h4AXSRY9cuk+g=="}' 2024-06-11T17:02:38.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9de08238-70ac-466b-8ab4-2c16a17aa6f8 -i td/osd-recovery-scrub/2/new.json 2024-06-11T17:02:39.299 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:02:39.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-11T17:02:39.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAug2hmAB8bNRAAobiNMcKY9h4AXSRY9cuk+g== --osd-uuid 9de08238-70ac-466b-8ab4-2c16a17aa6f8 2024-06-11T17:02:39.355 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.353+0000 7fe04ff097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:39.401 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.401+0000 7fe04ff097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:39.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.401+0000 7fe04ff097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:39.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.401+0000 7fe04ff097c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:02:39.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.401+0000 7fe04ff097c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:02:39.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.401+0000 7fe04ff097c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:02:39.479 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:39.481+0000 7fe04ff097c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-11T17:02:45.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-11T17:02:45.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:02:45.886 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:02:45.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:02:45.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:02:46.291 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:02:46.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:02:46.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-11T17:02:46.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:02:46.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:02:46.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:02:46.317 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:46.317+0000 7fd5880607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:46.320 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:46.321+0000 7fd5880607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:46.321+0000 7fd5880607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:46.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:02:46.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:02:46.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:02:46.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:02:46.647 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:02:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:02:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:02:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:02:46.845 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:46.845+0000 7fd5880607c0 -1 Falling back to public interface 2024-06-11T17:02:46.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:47.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:47.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:47.996 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:02:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:02:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:02:48.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:49.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:49.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:49.320 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:02:49.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:02:49.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:49.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:02:49.488 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:49.489+0000 7fd5880607c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:02:49.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:02:50.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:02:50.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:50.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:02:50.669 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:02:50.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:50.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:02:51.011 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3851488430,v1:127.0.0.1:6819/3851488430] [v2:127.0.0.1:6820/3851488430,v1:127.0.0.1:6821/3851488430] exists,up 9de08238-70ac-466b-8ab4-2c16a17aa6f8 2024-06-11T17:02:51.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:02:51.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:02:51.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:02:51.016 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:02:51.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:02:51.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:02:51.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:02:51.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de7d68a5-557d-4320-9248-1536708b8a95 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 de7d68a5-557d-4320-9248-1536708b8a95' 2024-06-11T17:02:51.019 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 de7d68a5-557d-4320-9248-1536708b8a95 2024-06-11T17:02:51.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:02:51.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7g2hm0uQxAhAAGSIaQc425nS8HfLXiOWYfQ== 2024-06-11T17:02:51.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7g2hm0uQxAhAAGSIaQc425nS8HfLXiOWYfQ=="}' 2024-06-11T17:02:51.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de7d68a5-557d-4320-9248-1536708b8a95 -i td/osd-recovery-scrub/3/new.json 2024-06-11T17:02:51.400 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:02:51.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-11T17:02:51.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQA7g2hm0uQxAhAAGSIaQc425nS8HfLXiOWYfQ== --osd-uuid de7d68a5-557d-4320-9248-1536708b8a95 2024-06-11T17:02:51.460 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.461+0000 7fb899b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:51.493 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.493+0000 7fb899b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:51.494 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.493+0000 7fb899b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:51.495 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.493+0000 7fb899b137c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:02:51.495 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.493+0000 7fb899b137c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:02:51.495 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.493+0000 7fb899b137c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:02:51.578 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:51.577+0000 7fb899b137c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-11T17:02:58.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-11T17:02:58.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:02:58.301 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:02:58.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:02:58.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:02:58.713 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:02:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:02:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:02:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:02:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:02:58.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-11T17:02:58.748 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:58.749+0000 7ff07f4807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:58.775 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:58.773+0000 7ff07f4807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:58.776 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:58.777+0000 7ff07f4807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:02:59.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:02:59.057 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:02:59.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:02:59.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:02:59.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:02:59.322 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:02:59.321+0000 7ff07f4807c0 -1 Falling back to public interface 2024-06-11T17:02:59.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:03:00.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:03:00.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:03:00.421 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:03:00.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:03:00.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:03:00.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:03:00.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:03:01.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:03:01.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:03:01.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:03:01.759 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:03:01.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:03:01.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:03:02.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:03:02.110 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:03:02.109+0000 7ff07f4807c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:03:03.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:03:03.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:03:03.108 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:03:03.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:03:03.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:03:03.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:03:03.529 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2196347513,v1:127.0.0.1:6827/2196347513] [v2:127.0.0.1:6828/2196347513,v1:127.0.0.1:6829/2196347513] exists,up de7d68a5-557d-4320-9248-1536708b8a95 2024-06-11T17:03:03.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:03:03.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:03:03.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:03:03.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-06-11T17:03:03.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-06-11T17:03:03.963 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'test' created 2024-06-11T17:03:03.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:03:04.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-06-11T17:03:04.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:03:04.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:03:04.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:03:04.988 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:03:04.989 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:03:04.989 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:03:04.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:03:04.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:03:04.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:03:05.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:03:05.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:03:05.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:03:05.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:03:05.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:03:05.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:03:05.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:03:05.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:03:05.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-06-11T17:03:05.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-06-11T17:03:05.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-06-11T17:03:05.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:03:05.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:03:05.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672967 2024-06-11T17:03:05.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672967 2024-06-11T17:03:05.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-06-11T17:03:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:03:05.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:03:05.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509445 2024-06-11T17:03:05.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509445 2024-06-11T17:03:05.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445' 2024-06-11T17:03:05.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:03:05.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:03:06.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345922 2024-06-11T17:03:06.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345922 2024-06-11T17:03:06.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445 3-85899345922' 2024-06-11T17:03:06.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:03:06.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-06-11T17:03:06.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:03:06.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:03:06.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-06-11T17:03:06.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:03:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-06-11T17:03:06.117 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836489 2024-06-11T17:03:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-11T17:03:06.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:03:06.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836489 2024-06-11T17:03:06.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:03:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:03:07.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:03:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836489 2024-06-11T17:03:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:03:07.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672967 2024-06-11T17:03:07.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:03:07.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:03:07.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672967 2024-06-11T17:03:07.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:03:07.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672967 2024-06-11T17:03:07.816 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672967 2024-06-11T17:03:07.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-11T17:03:07.816 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:03:08.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-11T17:03:08.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:03:08.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509445 2024-06-11T17:03:08.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:03:08.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:03:08.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509445 2024-06-11T17:03:08.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:03:08.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509445 2024-06-11T17:03:08.167 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509445 2024-06-11T17:03:08.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-11T17:03:08.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:03:08.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509445 2024-06-11T17:03:08.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:03:08.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345922 2024-06-11T17:03:08.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:03:08.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:03:08.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345922 2024-06-11T17:03:08.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:03:08.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345922 2024-06-11T17:03:08.517 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345922 2024-06-11T17:03:08.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-06-11T17:03:08.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:03:08.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345922 -lt 85899345922 2024-06-11T17:03:08.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:03:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:03:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:03:09.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:03:09.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:03:09.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:03:09.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:03:09.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:03:09.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:03:09.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:03:10.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:03:10.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:03:10.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:03:10.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-06-11T17:03:10.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-06-11T17:03:10.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-06-11T17:03:10.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-06-11T17:03:10.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-11T17:03:10.763 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:03:10.763 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:03:05.104538+0000 0'0 24:12 [1] 1 [1] 1 0'0 2024-06-11T17:03:03.903156+0000 0'0 2024-06-11T17:03:03.903156+0000 0 0 periodic scrub scheduled @ 2024-06-12T19:35:45.062140+0000 0 0 2024-06-11T17:03:10.763 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:03:10.763 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:03:10.763 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:03:10.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.36888 bs=1M count=50 2024-06-11T17:03:10.946 INFO:tasks.workunit.client.0.smithi012.stderr:50+0 records in 2024-06-11T17:03:10.946 INFO:tasks.workunit.client.0.smithi012.stderr:50+0 records out 2024-06-11T17:03:10.946 INFO:tasks.workunit.client.0.smithi012.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.165342 s, 317 MB/s 2024-06-11T17:03:10.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-06-11T17:03:10.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:10.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.36888 2024-06-11T17:03:12.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:12.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.36888 2024-06-11T17:03:14.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:14.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.36888 2024-06-11T17:03:16.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:16.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.36888 2024-06-11T17:03:17.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:17.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.36888 2024-06-11T17:03:19.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:19.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.36888 2024-06-11T17:03:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.36888 2024-06-11T17:03:23.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:23.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.36888 2024-06-11T17:03:24.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:24.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.36888 2024-06-11T17:03:26.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:26.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.36888 2024-06-11T17:03:28.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:28.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.36888 2024-06-11T17:03:29.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:29.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.36888 2024-06-11T17:03:31.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:31.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.36888 2024-06-11T17:03:33.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:33.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.36888 2024-06-11T17:03:35.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:35.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.36888 2024-06-11T17:03:36.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:36.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.36888 2024-06-11T17:03:38.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:38.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.36888 2024-06-11T17:03:40.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:40.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.36888 2024-06-11T17:03:42.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:42.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.36888 2024-06-11T17:03:44.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:44.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.36888 2024-06-11T17:03:46.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:46.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.36888 2024-06-11T17:03:47.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:47.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.36888 2024-06-11T17:03:49.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:49.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.36888 2024-06-11T17:03:51.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:51.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.36888 2024-06-11T17:03:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:52.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.36888 2024-06-11T17:03:54.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:54.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.36888 2024-06-11T17:03:56.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:56.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.36888 2024-06-11T17:03:58.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:58.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.36888 2024-06-11T17:03:59.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:03:59.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.36888 2024-06-11T17:04:01.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:01.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.36888 2024-06-11T17:04:03.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:03.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.36888 2024-06-11T17:04:04.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:04.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.36888 2024-06-11T17:04:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.36888 2024-06-11T17:04:08.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:08.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.36888 2024-06-11T17:04:10.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:10.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.36888 2024-06-11T17:04:11.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:11.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.36888 2024-06-11T17:04:13.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:13.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.36888 2024-06-11T17:04:14.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:14.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.36888 2024-06-11T17:04:16.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:16.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.36888 2024-06-11T17:04:18.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:18.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.36888 2024-06-11T17:04:20.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:20.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.36888 2024-06-11T17:04:21.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:21.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.36888 2024-06-11T17:04:23.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:23.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.36888 2024-06-11T17:04:25.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:25.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.36888 2024-06-11T17:04:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.36888 2024-06-11T17:04:28.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:28.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.36888 2024-06-11T17:04:30.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:30.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.36888 2024-06-11T17:04:31.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:31.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.36888 2024-06-11T17:04:33.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:33.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.36888 2024-06-11T17:04:35.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:35.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.36888 2024-06-11T17:04:37.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:37.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.36888 2024-06-11T17:04:38.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:38.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.36888 2024-06-11T17:04:40.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:40.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.36888 2024-06-11T17:04:42.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:42.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.36888 2024-06-11T17:04:43.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:43.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.36888 2024-06-11T17:04:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.36888 2024-06-11T17:04:47.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:47.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.36888 2024-06-11T17:04:48.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:48.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.36888 2024-06-11T17:04:50.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:50.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.36888 2024-06-11T17:04:52.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:52.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.36888 2024-06-11T17:04:54.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:54.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.36888 2024-06-11T17:04:55.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:55.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.36888 2024-06-11T17:04:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.36888 2024-06-11T17:04:59.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:04:59.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.36888 2024-06-11T17:05:01.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:01.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.36888 2024-06-11T17:05:02.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:02.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.36888 2024-06-11T17:05:04.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:04.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.36888 2024-06-11T17:05:06.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:06.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.36888 2024-06-11T17:05:08.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:08.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.36888 2024-06-11T17:05:10.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.36888 2024-06-11T17:05:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.36888 2024-06-11T17:05:13.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:13.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.36888 2024-06-11T17:05:15.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:15.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.36888 2024-06-11T17:05:17.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:17.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.36888 2024-06-11T17:05:19.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:19.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.36888 2024-06-11T17:05:21.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:21.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.36888 2024-06-11T17:05:22.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:22.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.36888 2024-06-11T17:05:24.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:24.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.36888 2024-06-11T17:05:26.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:26.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.36888 2024-06-11T17:05:28.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:28.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.36888 2024-06-11T17:05:30.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:30.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.36888 2024-06-11T17:05:31.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:31.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.36888 2024-06-11T17:05:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.36888 2024-06-11T17:05:35.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:35.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.36888 2024-06-11T17:05:37.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:37.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.36888 2024-06-11T17:05:38.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:38.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.36888 2024-06-11T17:05:40.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:40.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.36888 2024-06-11T17:05:42.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:42.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.36888 2024-06-11T17:05:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.36888 2024-06-11T17:05:46.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:46.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.36888 2024-06-11T17:05:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.36888 2024-06-11T17:05:49.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:49.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.36888 2024-06-11T17:05:51.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:51.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.36888 2024-06-11T17:05:53.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:53.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.36888 2024-06-11T17:05:55.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:55.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.36888 2024-06-11T17:05:56.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:56.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.36888 2024-06-11T17:05:58.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:05:58.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.36888 2024-06-11T17:06:00.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:06:00.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.36888 2024-06-11T17:06:02.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:06:02.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.36888 2024-06-11T17:06:04.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-11T17:06:04.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.36888 2024-06-11T17:06:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.36888 2024-06-11T17:06:05.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-06-11T17:06:06.399 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 4 2024-06-11T17:06:06.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-06-11T17:06:06.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-06-11T17:06:06.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-11T17:06:06.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-11T17:06:06.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-11T17:06:06.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-11T17:06:06.786 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:06.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-11T17:06:08.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-06-11T17:06:08.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-06-11T17:06:08.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-06-11T17:06:08.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-11T17:06:08.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-11T17:06:08.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-11T17:06:08.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-11T17:06:09.128 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:09.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-11T17:06:11.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-06-11T17:06:11.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-06-11T17:06:11.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-06-11T17:06:11.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-11T17:06:11.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-11T17:06:11.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-11T17:06:11.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-11T17:06:11.469 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-11T17:06:13.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-06-11T17:06:13.489 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-06-11T17:06:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-06-11T17:06:13.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-11T17:06:13.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-11T17:06:13.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-11T17:06:13.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-11T17:06:14.013 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:14.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-11T17:06:16.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-06-11T17:06:16.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-06-11T17:06:16.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-06-11T17:06:16.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-11T17:06:16.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-11T17:06:16.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-11T17:06:16.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-11T17:06:16.353 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:16.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-06-11T17:06:16.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-06-11T17:06:16.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-11T17:06:16.680 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:06:16.680 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-06-11T17:06:09.215192+0000 24'1300 27:1328 [1,0,2,3] 1 [1,0] 1 0'0 2024-06-11T17:03:03.903156+0000 0'0 2024-06-11T17:03:03.903156+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:06:16.680 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:06:16.680 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:06:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:06:16.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-06-11T17:06:26.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-06-11T17:06:26.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:06:26.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:06:26.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:06:26.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:06:26.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:06:32.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:06:32.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-06-11T17:06:32.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-06-11T17:06:32.022 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-06-11T17:06:32.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-06-11T17:06:32.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-11T17:06:32.027 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:13.035+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:17.155+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:19.191+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:20.187+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:28.039+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.028 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:30.043+0000 7f6a7d9eb640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-11T17:06:32.044 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:10.087+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.044 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:14.135+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.044 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:16.175+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.044 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:18.187+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.044 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:19.191+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:26.203+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:27.219+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:29.175+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:30.219+0000 7f9557c2a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-11T17:06:32.047 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:19.531+0000 7fd581014640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:11.091+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:13.071+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:20.055+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:22.035+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:24.031+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:25.055+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.051 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:06:31.095+0000 7ff078434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-11T17:06:32.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-06-11T17:06:32.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-06-11T17:06:32.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-06-11T17:06:32.052 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-06-11T17:06:32.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-06-11T17:06:32.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-11T17:06:32.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-11T17:06:32.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-06-11T17:06:32.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-06-11T17:06:32.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-11T17:06:32.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-11T17:06:32.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-06-11T17:06:32.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-06-11T17:06:32.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-11T17:06:32.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-11T17:06:32.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-06-11T17:06:32.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-06-11T17:06:32.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:32.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-11T17:06:32.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-11T17:06:32.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-06-11T17:06:32.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:06:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:06:32.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:06:32.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:06:32.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:06:32.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:06:32.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:06:32.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:06:32.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:06:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:06:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:06:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:06:32.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:06:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:06:32.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:06:32.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-11T17:06:32.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:06:32.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36888 2024-06-11T17:06:32.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:06:32.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:06:32.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-06-11T17:06:32.127 INFO:tasks.workunit.client.0.smithi012.stdout:TEST PASSED 2024-06-11T17:06:32.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-06-11T17:06:32.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-06-11T17:06:32.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.36888 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-11T17:06:32.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:06:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:06:32.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:06:32.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:06:32.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:06:32.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:06:32.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:06:32.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:06:32.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:06:32.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:06:32.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:06:32.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:06:32.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-11T17:06:32.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:06:32.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36888 2024-06-11T17:06:32.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:06:32.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:06:32.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-11T17:06:32.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:06:32.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36888 2024-06-11T17:06:32.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:06:32.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:06:32.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:06:32.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-11T17:06:32.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:06:32.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-11T17:06:32.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:06:32.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:06:32.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:06:32.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-11T17:06:32.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:06:32.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:06:32.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:06:32.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:06:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:06:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:32.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-11T17:06:32.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:06:32.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:06:32.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:06:32.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36888/ceph-mon.a.asok 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:06:32.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36888/ceph-mon.a.asok config get fsid 2024-06-11T17:06:32.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:06:32.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:06:32.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:06:32.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:06:32.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:06:32.954 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:06:32.954 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:06:32.954 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:06:32.955 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:06:32.955 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:32.955 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:32.955 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36888/ceph-mon.a.asok 2024-06-11T17:06:32.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:06:32.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36888/ceph-mon.a.asok config get mon_host 2024-06-11T17:06:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-06-11T17:06:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-11T17:06:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:06:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:06:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:06:33.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-11T17:06:33.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:06:33.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:06:33.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:06:33.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:06:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:06:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:33.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:33.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:06:33.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:06:34.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-11T17:06:34.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-11T17:06:34.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-11T17:06:34.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-11T17:06:34.021 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-06-11T17:06:34.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-06-11T17:06:34.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:34.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:06:34.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:06:34.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:06:34.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:06:34.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-11T17:06:34.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:06:34.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab 2024-06-11T17:06:34.029 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab 2024-06-11T17:06:34.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab' 2024-06-11T17:06:34.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:06:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAahGhmM5HZAhAAPXVTbymsAawbe3Bx9ARO7Q== 2024-06-11T17:06:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAahGhmM5HZAhAAPXVTbymsAawbe3Bx9ARO7Q=="}' 2024-06-11T17:06:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab -i td/osd-recovery-scrub/0/new.json 2024-06-11T17:06:34.308 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:06:34.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-11T17:06:34.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAahGhmM5HZAhAAPXVTbymsAawbe3Bx9ARO7Q== --osd-uuid 21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab 2024-06-11T17:06:34.357 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.355+0000 7f35930047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:34.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.371+0000 7f35930047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.375+0000 7f35930047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:34.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.375+0000 7f35930047c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:06:34.377 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.375+0000 7f35930047c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:06:34.377 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.375+0000 7f35930047c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-11T17:06:34.467 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:34.463+0000 7f35930047c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-11T17:06:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-11T17:06:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:06:41.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:06:41.123 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:06:41.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:06:41.536 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:06:41.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:06:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:06:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:06:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:06:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:06:41.576 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:41.575+0000 7f7ed5c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:41.607 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:41.603+0000 7f7ed5c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:41.608 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:41.607+0000 7f7ed5c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:06:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:06:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:06:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:06:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:06:41.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:41.886 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:06:41.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:06:41.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:41.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:06:42.154 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:42.151+0000 7f7ed5c197c0 -1 Falling back to public interface 2024-06-11T17:06:42.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:43.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:43.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:43.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:06:43.232 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:06:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:06:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:44.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:44.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:44.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:06:44.567 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:06:44.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:44.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:06:44.724 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:44.723+0000 7f7ed5c197c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:06:44.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:45.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:45.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:45.941 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:06:45.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:06:45.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:45.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:06:46.280 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/821801098,v1:127.0.0.1:6803/821801098] [v2:127.0.0.1:6804/821801098,v1:127.0.0.1:6805/821801098] exists,up 21ab3d72-3fdc-4d9d-81c5-eb8efa2e58ab 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:06:46.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:06:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:06:46.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:06:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-11T17:06:46.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:06:46.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6494ab3b-f274-4038-a314-5e326a8fecfa 2024-06-11T17:06:46.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6494ab3b-f274-4038-a314-5e326a8fecfa' 2024-06-11T17:06:46.287 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 6494ab3b-f274-4038-a314-5e326a8fecfa 2024-06-11T17:06:46.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:06:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmhGhmWH2JEhAAr6tju1TNn+aLl0N+OwyXUA== 2024-06-11T17:06:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmhGhmWH2JEhAAr6tju1TNn+aLl0N+OwyXUA=="}' 2024-06-11T17:06:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6494ab3b-f274-4038-a314-5e326a8fecfa -i td/osd-recovery-scrub/1/new.json 2024-06-11T17:06:46.684 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:06:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-11T17:06:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAmhGhmWH2JEhAAr6tju1TNn+aLl0N+OwyXUA== --osd-uuid 6494ab3b-f274-4038-a314-5e326a8fecfa 2024-06-11T17:06:46.742 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.739+0000 7fc0d04a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:46.802 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.799+0000 7fc0d04a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:46.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.803+0000 7fc0d04a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:46.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.803+0000 7fc0d04a97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:06:46.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.803+0000 7fc0d04a97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:06:46.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.803+0000 7fc0d04a97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-11T17:06:46.881 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:46.883+0000 7fc0d04a97c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-11T17:06:53.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-11T17:06:53.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:06:53.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:06:53.584 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:06:53.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:06:54.000 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:06:54.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:06:54.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:06:54.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:06:54.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:06:54.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:06:54.040 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:54.039+0000 7fe71b1ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:54.066 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:54.063+0000 7fe71b1ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:54.068 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:54.067+0000 7fe71b1ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:06:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:54.355 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:06:54.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:06:54.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:54.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:06:54.629 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:54.627+0000 7fe71b1ff7c0 -1 Falling back to public interface 2024-06-11T17:06:54.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:55.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:55.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:55.721 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:06:55.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:06:55.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:55.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:06:56.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:57.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:57.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:57.057 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:06:57.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:06:57.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:57.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:06:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:06:57.594 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:57.579+0000 7fe71b1ff7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:06:58.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:06:58.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:06:58.495 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:06:58.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:06:58.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:06:58.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:06:58.833 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2587935794,v1:127.0.0.1:6811/2587935794] [v2:127.0.0.1:6812/2587935794,v1:127.0.0.1:6813/2587935794] exists,up 6494ab3b-f274-4038-a314-5e326a8fecfa 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:06:58.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:06:58.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:06:58.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:06:58.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:06:58.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:06:58.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:06:58.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:06:58.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:58.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-11T17:06:58.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:06:58.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0dcad7f-b860-4147-811d-81443a3a4a58 2024-06-11T17:06:58.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b0dcad7f-b860-4147-811d-81443a3a4a58' 2024-06-11T17:06:58.840 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 b0dcad7f-b860-4147-811d-81443a3a4a58 2024-06-11T17:06:58.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:06:58.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyhGhm8KV6MxAAD9n1I8U8diTWloivonivAg== 2024-06-11T17:06:58.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyhGhm8KV6MxAAD9n1I8U8diTWloivonivAg=="}' 2024-06-11T17:06:58.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0dcad7f-b860-4147-811d-81443a3a4a58 -i td/osd-recovery-scrub/2/new.json 2024-06-11T17:06:59.287 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:06:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-11T17:06:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAyhGhm8KV6MxAAD9n1I8U8diTWloivonivAg== --osd-uuid b0dcad7f-b860-4147-811d-81443a3a4a58 2024-06-11T17:06:59.345 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.343+0000 7f38a64857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:59.389 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.387+0000 7f38a64857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:59.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.387+0000 7f38a64857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:06:59.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.391+0000 7f38a64857c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:06:59.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.391+0000 7f38a64857c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:06:59.392 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.391+0000 7f38a64857c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-11T17:06:59.462 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:06:59.459+0000 7f38a64857c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-11T17:07:06.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-11T17:07:06.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:07:06.286 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:07:06.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:07:06.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:07:06.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:07:06.701 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:07:06.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:07:06.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:07:06.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:06.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:07:06.740 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:06.739+0000 7f68ccf2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:06.772 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:06.771+0000 7f68ccf2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:06.774 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:06.771+0000 7f68ccf2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:07.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:07:07.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:07.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:07:07.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:07.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:07:07.345 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:07.343+0000 7f68ccf2d7c0 -1 Falling back to public interface 2024-06-11T17:07:07.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:08.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:08.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:08.402 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:07:08.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:07:08.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:08.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:07:08.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:09.749 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:07:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:07:09.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:09.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:07:09.975 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:09.971+0000 7f68ccf2d7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:07:10.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:11.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:11.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:11.097 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:07:11.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:07:11.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:11.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:07:11.481 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3352457097,v1:127.0.0.1:6819/3352457097] [v2:127.0.0.1:6820/3352457097,v1:127.0.0.1:6821/3352457097] exists,up b0dcad7f-b860-4147-811d-81443a3a4a58 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:11.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:07:11.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:07:11.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:07:11.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-11T17:07:11.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:07:11.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba2b04eb-5e66-4336-af36-fc526efd2146 2024-06-11T17:07:11.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ba2b04eb-5e66-4336-af36-fc526efd2146' 2024-06-11T17:07:11.488 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 ba2b04eb-5e66-4336-af36-fc526efd2146 2024-06-11T17:07:11.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:07:11.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/hGhmgJckHhAAIduzS8CIcz4jJ8AeCIhI/A== 2024-06-11T17:07:11.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/hGhmgJckHhAAIduzS8CIcz4jJ8AeCIhI/A=="}' 2024-06-11T17:07:11.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba2b04eb-5e66-4336-af36-fc526efd2146 -i td/osd-recovery-scrub/3/new.json 2024-06-11T17:07:11.893 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:07:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-11T17:07:11.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQA/hGhmgJckHhAAIduzS8CIcz4jJ8AeCIhI/A== --osd-uuid ba2b04eb-5e66-4336-af36-fc526efd2146 2024-06-11T17:07:11.951 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.947+0000 7f6f836717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:11.995 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.991+0000 7f6f836717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:11.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.995+0000 7f6f836717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:11.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.995+0000 7f6f836717c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:07:11.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.995+0000 7f6f836717c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:07:11.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:11.995+0000 7f6f836717c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-11T17:07:12.070 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:12.067+0000 7f6f836717c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-11T17:07:18.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-11T17:07:18.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:07:18.766 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:07:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:07:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:07:19.206 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:07:19.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:07:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:07:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:07:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:07:19.245 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:19.243+0000 7fc0f69647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:19.277 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:19.275+0000 7fc0f69647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:19.278 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:19.275+0000 7fc0f69647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:19.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:19.564 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:07:19.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:07:19.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:07:19.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:19.842 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:19.839+0000 7fc0f69647c0 -1 Falling back to public interface 2024-06-11T17:07:19.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:20.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:20.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:20.931 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:07:20.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:07:20.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:20.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:07:21.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:22.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:22.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:22.272 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:07:22.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:07:22.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:22.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:07:22.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:22.403+0000 7fc0f69647c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:07:22.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:23.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:23.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:07:23.674 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:07:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:07:24.010 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1783022657,v1:127.0.0.1:6827/1783022657] [v2:127.0.0.1:6828/1783022657,v1:127.0.0.1:6829/1783022657] exists,up ba2b04eb-5e66-4336-af36-fc526efd2146 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-11T17:07:24.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:07:24.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:07:24.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:07:24.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6f03115-9b1a-4034-b51f-6af8b226eb08 2024-06-11T17:07:24.018 INFO:tasks.workunit.client.0.smithi012.stdout:add osd4 c6f03115-9b1a-4034-b51f-6af8b226eb08 2024-06-11T17:07:24.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c6f03115-9b1a-4034-b51f-6af8b226eb08' 2024-06-11T17:07:24.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:07:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMhGhm1CIKAhAArd3if6l67FRqYAkhFH36Ew== 2024-06-11T17:07:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMhGhm1CIKAhAArd3if6l67FRqYAkhFH36Ew=="}' 2024-06-11T17:07:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6f03115-9b1a-4034-b51f-6af8b226eb08 -i td/osd-recovery-scrub/4/new.json 2024-06-11T17:07:24.424 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:07:24.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-06-11T17:07:24.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBMhGhm1CIKAhAArd3if6l67FRqYAkhFH36Ew== --osd-uuid c6f03115-9b1a-4034-b51f-6af8b226eb08 2024-06-11T17:07:24.482 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.479+0000 7f4b1aede7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:24.525 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.523+0000 7f4b1aede7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:24.527 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.523+0000 7f4b1aede7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:24.528 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.527+0000 7f4b1aede7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-11T17:07:24.528 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.527+0000 7f4b1aede7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-11T17:07:24.528 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.527+0000 7f4b1aede7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-11T17:07:24.597 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:24.595+0000 7f4b1aede7c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-06-11T17:07:31.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-06-11T17:07:31.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:07:31.500 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd4 key to auth repository 2024-06-11T17:07:31.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-11T17:07:31.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:07:31.920 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.4 2024-06-11T17:07:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-11T17:07:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:07:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:07:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:07:31.961 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:31.959+0000 7f8b83cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:31.984 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:31.983+0000 7f8b83cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:31.986 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:31.983+0000 7f8b83cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:32.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:07:32.262 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:07:32.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:32.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:07:32.526 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:32.523+0000 7f8b83cc17c0 -1 Falling back to public interface 2024-06-11T17:07:32.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:33.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:33.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:33.622 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:07:33.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:07:33.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:33.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:07:33.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:34.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:34.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:34.958 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:07:34.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:07:34.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:34.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:07:35.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:35.311+0000 7f8b83cc17c0 -1 osd.4 0 log_to_monitors true 2024-06-11T17:07:35.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:36.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:36.337 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:07:36.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:36.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:07:36.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:36.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:07:36.676 INFO:tasks.workunit.client.0.smithi012.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/754741585,v1:127.0.0.1:6835/754741585] [v2:127.0.0.1:6836/754741585,v1:127.0.0.1:6837/754741585] exists,up c6f03115-9b1a-4034-b51f-6af8b226eb08 2024-06-11T17:07:36.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:07:36.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:07:36.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:07:36.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:07:36.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-06-11T17:07:36.683 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:07:36.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d765b645-32f1-41e1-bdda-530f768a8ac4 2024-06-11T17:07:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:add osd5 d765b645-32f1-41e1-bdda-530f768a8ac4 2024-06-11T17:07:36.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 d765b645-32f1-41e1-bdda-530f768a8ac4' 2024-06-11T17:07:36.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:07:36.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYhGhmwNfNKRAACbbLb7tBM0viPnmgqoqD9A== 2024-06-11T17:07:36.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYhGhmwNfNKRAACbbLb7tBM0viPnmgqoqD9A=="}' 2024-06-11T17:07:36.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d765b645-32f1-41e1-bdda-530f768a8ac4 -i td/osd-recovery-scrub/5/new.json 2024-06-11T17:07:37.316 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:07:37.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-06-11T17:07:37.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBYhGhmwNfNKRAACbbLb7tBM0viPnmgqoqD9A== --osd-uuid d765b645-32f1-41e1-bdda-530f768a8ac4 2024-06-11T17:07:37.386 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.383+0000 7fd785b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:37.444 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.443+0000 7fd785b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:37.445 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.443+0000 7fd785b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:37.446 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.443+0000 7fd785b787c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-11T17:07:37.446 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.443+0000 7fd785b787c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-11T17:07:37.446 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.443+0000 7fd785b787c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-11T17:07:37.542 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:37.539+0000 7fd785b787c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-06-11T17:07:44.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-06-11T17:07:44.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:07:44.594 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd5 key to auth repository 2024-06-11T17:07:44.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-11T17:07:44.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:07:45.038 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.5 2024-06-11T17:07:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-11T17:07:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:07:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:07:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:07:45.076 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:45.075+0000 7f9ed935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:45.103 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:45.103+0000 7f9ed935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:45.103+0000 7f9ed935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:45.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-11T17:07:45.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:07:45.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:07:45.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:07:45.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:45.659+0000 7f9ed935e7c0 -1 Falling back to public interface 2024-06-11T17:07:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:46.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:46.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:46.740 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:07:46.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:07:46.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:46.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:07:47.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:48.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:48.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:48.082 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:07:48.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:07:48.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:48.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:07:48.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:48.311+0000 7f9ed935e7c0 -1 osd.5 0 log_to_monitors true 2024-06-11T17:07:48.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:07:49.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:07:49.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:49.480 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:07:49.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:07:49.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:49.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:07:49.820 INFO:tasks.workunit.client.0.smithi012.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/887779168,v1:127.0.0.1:6843/887779168] [v2:127.0.0.1:6844/887779168,v1:127.0.0.1:6845/887779168] exists,up d765b645-32f1-41e1-bdda-530f768a8ac4 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:49.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:07:49.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:07:49.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:49.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:07:49.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-06-11T17:07:49.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:07:49.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=801e156c-6d6a-4801-9351-fbc655b11505 2024-06-11T17:07:49.827 INFO:tasks.workunit.client.0.smithi012.stdout:add osd6 801e156c-6d6a-4801-9351-fbc655b11505 2024-06-11T17:07:49.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 801e156c-6d6a-4801-9351-fbc655b11505' 2024-06-11T17:07:49.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:07:49.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlhGhmAYd/MhAA0pVUXkKCCuES4GELUvyHvQ== 2024-06-11T17:07:49.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlhGhmAYd/MhAA0pVUXkKCCuES4GELUvyHvQ=="}' 2024-06-11T17:07:49.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 801e156c-6d6a-4801-9351-fbc655b11505 -i td/osd-recovery-scrub/6/new.json 2024-06-11T17:07:50.269 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:07:50.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-06-11T17:07:50.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBlhGhmAYd/MhAA0pVUXkKCCuES4GELUvyHvQ== --osd-uuid 801e156c-6d6a-4801-9351-fbc655b11505 2024-06-11T17:07:50.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.330+0000 7f95d40df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:50.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.374+0000 7f95d40df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:50.381 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.378+0000 7f95d40df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:50.382 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.378+0000 7f95d40df7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-11T17:07:50.382 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.378+0000 7f95d40df7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-11T17:07:50.382 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.378+0000 7f95d40df7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-11T17:07:50.450 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:50.446+0000 7f95d40df7c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-06-11T17:07:57.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-06-11T17:07:57.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:07:57.971 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd6 key to auth repository 2024-06-11T17:07:57.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-11T17:07:57.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:07:58.376 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.6 2024-06-11T17:07:58.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-11T17:07:58.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:07:58.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:07:58.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:07:58.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:07:58.415 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:58.410+0000 7f1b1174c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:58.441 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:58.438+0000 7f1b1174c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:58.443 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:58.438+0000 7f1b1174c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:07:58.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:07:58.715 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:07:58.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:07:58.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:07:58.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:07:58.998 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:07:58.994+0000 7f1b1174c7c0 -1 Falling back to public interface 2024-06-11T17:07:59.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:00.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:00.065 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:08:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:08:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:08:00.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:01.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:01.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:01.405 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:08:01.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:08:01.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:01.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:08:01.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:01.746 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:01.742+0000 7f1b1174c7c0 -1 osd.6 0 log_to_monitors true 2024-06-11T17:08:02.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:02.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:02.741 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:08:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:08:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:08:03.138 INFO:tasks.workunit.client.0.smithi012.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3137679501,v1:127.0.0.1:6851/3137679501] [v2:127.0.0.1:6852/3137679501,v1:127.0.0.1:6853/3137679501] exists,up 801e156c-6d6a-4801-9351-fbc655b11505 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-11T17:08:03.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:08:03.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:08:03.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:08:03.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:08:03.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:08:03.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-11T17:08:03.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-06-11T17:08:03.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:08:03.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=015b312b-3c4d-462d-bf19-3c3c8bbcb50c 2024-06-11T17:08:03.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 015b312b-3c4d-462d-bf19-3c3c8bbcb50c' 2024-06-11T17:08:03.147 INFO:tasks.workunit.client.0.smithi012.stdout:add osd7 015b312b-3c4d-462d-bf19-3c3c8bbcb50c 2024-06-11T17:08:03.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:08:03.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzhGhmYkgOChAAZn5bNHLC9nrziuqoSrxRmg== 2024-06-11T17:08:03.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzhGhmYkgOChAAZn5bNHLC9nrziuqoSrxRmg=="}' 2024-06-11T17:08:03.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 015b312b-3c4d-462d-bf19-3c3c8bbcb50c -i td/osd-recovery-scrub/7/new.json 2024-06-11T17:08:03.580 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:08:03.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-06-11T17:08:03.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBzhGhmYkgOChAAZn5bNHLC9nrziuqoSrxRmg== --osd-uuid 015b312b-3c4d-462d-bf19-3c3c8bbcb50c 2024-06-11T17:08:03.641 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.638+0000 7f59d26527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:03.647 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.642+0000 7f59d26527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:03.649 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.646+0000 7f59d26527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:03.650 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.646+0000 7f59d26527c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-11T17:08:03.650 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.646+0000 7f59d26527c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-11T17:08:03.650 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.646+0000 7f59d26527c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-11T17:08:03.781 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:03.782+0000 7f59d26527c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-06-11T17:08:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-06-11T17:08:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:08:11.030 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd7 key to auth repository 2024-06-11T17:08:11.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-11T17:08:11.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:08:11.493 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.7 2024-06-11T17:08:11.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-11T17:08:11.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:08:11.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:08:11.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:08:11.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=a762c787-4212-403c-a9c5-666daafa43d7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36888/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-11T17:08:11.532 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:11.534+0000 7fa5721457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:11.546 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:11.546+0000 7fa5721457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:11.548 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:11.550+0000 7fa5721457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:08:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:11.830 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:08:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:08:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:08:12.094 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:12.094+0000 7fa5721457c0 -1 Falling back to public interface 2024-06-11T17:08:12.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:13.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:13.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:13.178 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:08:13.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:08:13.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:13.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:08:13.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:14.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:14.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:14.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:08:14.517 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:08:14.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:14.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:08:14.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:08:14.778+0000 7fa5721457c0 -1 osd.7 0 log_to_monitors true 2024-06-11T17:08:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:08:15.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:08:15.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:08:15.882 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:08:15.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:08:15.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:08:15.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:08:16.258 INFO:tasks.workunit.client.0.smithi012.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1685188447,v1:127.0.0.1:6859/1685188447] [v2:127.0.0.1:6860/1685188447,v1:127.0.0.1:6861/1685188447] exists,up 015b312b-3c4d-462d-bf19-3c3c8bbcb50c 2024-06-11T17:08:16.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:08:16.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:08:16.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:08:16.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-06-11T17:08:16.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-06-11T17:08:16.780 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'test' created 2024-06-11T17:08:16.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:08:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-06-11T17:08:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:08:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:08:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:08:17.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:08:17.810 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:08:17.811 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:08:17.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:08:17.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:08:17.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:08:17.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:08:18.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:08:18.447 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:08:18.447 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:08:18.447 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:08:18.447 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr:7' 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:18.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:08:18.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-06-11T17:08:18.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-06-11T17:08:18.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-06-11T17:08:18.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:18.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:08:18.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-06-11T17:08:19.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-06-11T17:08:19.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-06-11T17:08:19.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:19.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:08:19.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-06-11T17:08:19.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-06-11T17:08:19.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455' 2024-06-11T17:08:19.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:19.379 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:08:19.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-06-11T17:08:19.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-06-11T17:08:19.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933' 2024-06-11T17:08:19.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:19.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:08:19.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182410 2024-06-11T17:08:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182410 2024-06-11T17:08:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933 4-107374182410' 2024-06-11T17:08:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:19.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:08:19.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018887 2024-06-11T17:08:19.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018887 2024-06-11T17:08:19.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933 4-107374182410 5-128849018887' 2024-06-11T17:08:19.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:19.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:08:20.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855365 2024-06-11T17:08:20.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855365 2024-06-11T17:08:20.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933 4-107374182410 5-128849018887 6-150323855365' 2024-06-11T17:08:20.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:08:20.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:08:20.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691842 2024-06-11T17:08:20.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691842 2024-06-11T17:08:20.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933 4-107374182410 5-128849018887 6-150323855365 7-171798691842' 2024-06-11T17:08:20.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:20.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-06-11T17:08:20.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:20.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:08:20.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-06-11T17:08:20.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:20.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-06-11T17:08:20.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-06-11T17:08:20.337 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836500 2024-06-11T17:08:20.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:08:20.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836500 2024-06-11T17:08:20.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:20.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-06-11T17:08:20.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:20.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:08:20.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-06-11T17:08:20.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:20.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-06-11T17:08:20.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-11T17:08:20.707 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672977 2024-06-11T17:08:20.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:08:21.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672977 2024-06-11T17:08:21.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:08:22.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:08:22.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:08:22.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672977 2024-06-11T17:08:22.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:22.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:22.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-06-11T17:08:22.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:08:22.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-06-11T17:08:22.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:22.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-06-11T17:08:22.437 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509455 2024-06-11T17:08:22.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-11T17:08:22.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:08:22.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-11T17:08:22.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:22.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345933 2024-06-11T17:08:22.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:22.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:08:22.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345933 2024-06-11T17:08:22.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:22.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-06-11T17:08:22.825 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345933 2024-06-11T17:08:22.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-06-11T17:08:22.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:08:23.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-06-11T17:08:23.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:23.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182410 2024-06-11T17:08:23.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:23.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:08:23.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182410 2024-06-11T17:08:23.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:23.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182410 2024-06-11T17:08:23.165 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.4 seq 107374182410 2024-06-11T17:08:23.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-06-11T17:08:23.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:08:23.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182411 -lt 107374182410 2024-06-11T17:08:23.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:23.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018887 2024-06-11T17:08:23.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:23.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:08:23.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018887 2024-06-11T17:08:23.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:23.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018887 2024-06-11T17:08:23.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-06-11T17:08:23.518 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.5 seq 128849018887 2024-06-11T17:08:23.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:08:23.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018887 2024-06-11T17:08:23.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:23.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855365 2024-06-11T17:08:23.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:23.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:08:23.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855365 2024-06-11T17:08:23.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:23.973 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.6 seq 150323855365 2024-06-11T17:08:23.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855365 2024-06-11T17:08:23.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-06-11T17:08:23.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:08:24.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-11T17:08:24.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:08:24.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:08:24.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691842 2024-06-11T17:08:24.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:08:24.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691842 2024-06-11T17:08:24.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:08:24.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691842 2024-06-11T17:08:24.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-06-11T17:08:24.326 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.7 seq 171798691842 2024-06-11T17:08:24.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:08:24.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691843 -lt 171798691842 2024-06-11T17:08:24.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:08:24.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:08:24.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:08:25.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:08:25.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:08:25.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:08:25.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-11T17:08:25.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:08:25.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:08:25.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:08:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-11T17:08:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:08:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:08:25.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-06-11T17:08:25.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-06-11T17:08:25.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-06-11T17:08:26.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-06-11T17:08:26.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.36888 bs=1M count=50 2024-06-11T17:08:26.307 INFO:tasks.workunit.client.0.smithi012.stderr:50+0 records in 2024-06-11T17:08:26.307 INFO:tasks.workunit.client.0.smithi012.stderr:50+0 records out 2024-06-11T17:08:26.307 INFO:tasks.workunit.client.0.smithi012.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.165919 s, 316 MB/s 2024-06-11T17:08:26.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-06-11T17:08:26.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:26.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.36888 2024-06-11T17:08:28.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:28.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.36888 2024-06-11T17:08:30.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:30.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.36888 2024-06-11T17:08:32.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:32.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.36888 2024-06-11T17:08:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.36888 2024-06-11T17:08:35.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:35.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.36888 2024-06-11T17:08:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.36888 2024-06-11T17:08:39.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:39.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.36888 2024-06-11T17:08:41.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:41.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.36888 2024-06-11T17:08:43.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:43.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.36888 2024-06-11T17:08:45.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:45.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.36888 2024-06-11T17:08:47.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:47.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.36888 2024-06-11T17:08:49.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:49.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.36888 2024-06-11T17:08:51.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:51.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.36888 2024-06-11T17:08:52.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:52.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.36888 2024-06-11T17:08:54.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:54.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.36888 2024-06-11T17:08:56.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:56.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.36888 2024-06-11T17:08:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.36888 2024-06-11T17:08:59.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:08:59.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.36888 2024-06-11T17:09:01.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:01.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.36888 2024-06-11T17:09:03.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:03.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.36888 2024-06-11T17:09:05.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:05.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.36888 2024-06-11T17:09:06.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:06.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.36888 2024-06-11T17:09:08.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:08.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.36888 2024-06-11T17:09:10.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:10.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.36888 2024-06-11T17:09:12.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:12.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.36888 2024-06-11T17:09:13.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:13.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.36888 2024-06-11T17:09:15.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:15.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.36888 2024-06-11T17:09:17.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:17.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.36888 2024-06-11T17:09:19.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:19.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.36888 2024-06-11T17:09:21.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:21.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.36888 2024-06-11T17:09:22.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:22.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.36888 2024-06-11T17:09:24.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:24.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.36888 2024-06-11T17:09:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:26.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.36888 2024-06-11T17:09:28.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:28.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.36888 2024-06-11T17:09:30.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:30.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.36888 2024-06-11T17:09:32.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:32.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.36888 2024-06-11T17:09:34.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:34.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.36888 2024-06-11T17:09:35.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:35.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.36888 2024-06-11T17:09:37.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-11T17:09:37.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.36888 2024-06-11T17:09:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.36888 2024-06-11T17:09:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-06-11T17:09:39.865 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 3 2024-06-11T17:09:39.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-11T17:08:18.742157+0000 50'26 50:64 [5] 5 [5] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.373862+0000 50'13 50:49 [3] 3 [3] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.1d 2 0 0 0 0 98566144 0 0 24 0 24 active+clean 2024-06-11T17:08:18.386554+0000 50'24 50:48 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.385810+0000 50'13 50:40 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.236526+0000 50'13 50:40 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:18.385096+0000 0'0 50:27 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-11T17:08:18.589500+0000 50'26 50:54 [5] 5 [5] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.206 INFO:tasks.workunit.client.0.smithi012.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.227955+0000 50'13 50:40 [6] 6 [6] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.226723+0000 50'13 50:42 [6] 6 [6] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-11T17:08:19.226788+0000 50'39 50:66 [4] 4 [4] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-11T17:08:19.373672+0000 50'52 50:80 [3] 3 [3] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-11T17:08:19.226696+0000 50'52 50:80 [4] 4 [4] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.589376+0000 50'13 50:41 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:19.226504+0000 0'0 50:24 [3] 3 [3] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.741644+0000 50'13 50:51 [5] 5 [5] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.742983+0000 50'13 50:35 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.202082+0000 50'13 50:47 [7] 7 [7] 7 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.201924+0000 50'13 50:35 [7] 7 [7] 7 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.386225+0000 50'13 50:39 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.226776+0000 50'13 50:38 [6] 6 [6] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.207 INFO:tasks.workunit.client.0.smithi012.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.741899+0000 50'13 50:39 [5] 5 [5] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.741266+0000 50'13 50:36 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-11T17:08:18.742282+0000 50'26 50:49 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.226787+0000 50'13 50:36 [4] 4 [4] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:18.068573+0000 0'0 50:25 [2] 2 [2] 2 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-11T17:08:18.386792+0000 50'39 50:64 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:19.228388+0000 0'0 50:24 [6] 6 [6] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.742655+0000 50'13 50:35 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:18.740916+0000 0'0 50:25 [5] 5 [5] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:19.373755+0000 50'13 50:35 [3] 3 [3] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:08:18.387238+0000 0'0 50:24 [1] 1 [1] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-11T17:08:18.741516+0000 50'13 50:35 [0] 0 [0] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:09:40.208 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:09:40.209 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:40.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-06-11T17:09:40.364 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:09:40.364 INFO:tasks.workunit.client.0.smithi012.stdout: "osd_recovery_max_active": "0" 2024-06-11T17:09:40.364 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:09:40.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-06-11T17:09:40.683 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:09:40.684 INFO:tasks.workunit.client.0.smithi012.stdout: "osd_recovery_max_active_hdd": "3" 2024-06-11T17:09:40.684 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:09:40.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-06-11T17:09:40.978 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:09:40.978 INFO:tasks.workunit.client.0.smithi012.stdout: "osd_recovery_max_active_ssd": "10" 2024-06-11T17:09:40.978 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:09:41.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-06-11T17:09:41.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-11T17:09:41.006 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-11T17:09:41.006 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-11T17:09:41.006 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-11T17:09:41.007 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-11T17:09:41.430 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:41.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-06-11T17:09:41.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-11T17:09:43.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-06-11T17:09:43.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-06-11T17:09:43.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-06-11T17:09:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-11T17:09:43.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-11T17:09:43.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-11T17:09:43.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-11T17:09:43.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-11T17:09:43.770 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:43.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-06-11T17:09:43.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-11T17:09:45.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-06-11T17:09:45.791 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-06-11T17:09:45.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-06-11T17:09:45.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-11T17:09:45.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-11T17:09:45.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-11T17:09:45.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-11T17:09:45.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-11T17:09:46.111 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:46.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-06-11T17:09:46.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-11T17:09:48.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-06-11T17:09:48.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-06-11T17:09:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-06-11T17:09:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-11T17:09:48.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-11T17:09:48.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-11T17:09:48.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-11T17:09:48.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-11T17:09:48.453 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:48.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-06-11T17:09:48.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-06-11T17:09:48.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-11T17:09:48.782 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:09:48.782 INFO:tasks.workunit.client.0.smithi012.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-11T17:09:42.028661+0000 50'26 53:75 [5,0,2] 5 [5,0,2] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.782 INFO:tasks.workunit.client.0.smithi012.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.095223+0000 50'13 53:59 [3,0,7] 3 [3,0,7] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-11T17:09:42.272344+0000 50'26 53:59 [1,3,0] 1 [1,3,0] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.473726+0000 50'13 53:49 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.438642+0000 50'13 53:49 [1,6,2] 1 [1,6,2] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:41.926038+0000 0'0 53:38 [1,4,5] 1 [1,4,5] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-11T17:09:42.164279+0000 50'26 53:63 [5,4,3] 5 [5,4,3] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-06-11T17:09:41.773976+0000 50'13 53:51 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:41.972973+0000 50'13 53:52 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-06-11T17:09:42.213512+0000 50'39 53:76 [4,2,6] 4 [4,2,6] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-06-11T17:09:42.438792+0000 50'52 53:89 [3,0,6] 3 [3,0,6] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-06-11T17:09:42.162816+0000 50'52 53:90 [4,1,2] 4 [4,1,2] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.473516+0000 50'13 53:51 [0,7,1] 0 [0,7,1] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.783 INFO:tasks.workunit.client.0.smithi012.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:41.805812+0000 0'0 53:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.270361+0000 50'13 53:62 [5,0,4] 5 [5,0,4] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.164575+0000 50'13 53:44 [0,3,4] 0 [0,3,4] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.473449+0000 50'13 53:58 [7,0,2] 7 [7,0,2] 7 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.580098+0000 50'13 53:45 [7,4,2] 7 [7,4,2] 7 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:41.974306+0000 50'13 53:48 [1,4,2] 1 [1,4,2] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.028892+0000 50'13 53:48 [6,0,3] 6 [6,0,3] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.441204+0000 50'13 53:49 [5,7,6] 5 [5,7,6] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.270563+0000 50'13 53:46 [0,1,3] 0 [0,1,3] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-11T17:09:42.438886+0000 50'26 53:58 [0,1,6] 0 [0,1,6] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.213832+0000 50'13 53:45 [4,6,3] 4 [4,6,3] 4 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:42.270867+0000 0'0 53:36 [2,0,4] 2 [2,0,4] 2 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.784 INFO:tasks.workunit.client.0.smithi012.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-06-11T17:09:41.925887+0000 50'39 53:73 [1,5,3] 1 [1,5,3] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:41.771572+0000 0'0 53:36 [6,0,7] 6 [6,0,7] 6 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.380521+0000 50'13 53:45 [0,4,5] 0 [0,4,5] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:41.767735+0000 0'0 53:36 [5,2,3] 5 [5,2,3] 5 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-11T17:09:42.270617+0000 50'13 53:44 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-11T17:09:42.095289+0000 0'0 53:35 [1,7,4] 1 [1,7,4] 1 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:08:16.654881+0000 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-06-11T17:09:42.030943+0000 50'13 53:45 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-11T17:08:16.654881+0000 0'0 2024-06-11T17:08:16.654881+0000 0 0 no scrub is scheduled 0 0 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:09:48.785 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:09:48.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-06-11T17:09:48.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-06-11T17:09:48.799 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-06-11T17:09:48.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-06-11T17:09:48.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58425"' 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58425' 2024-06-11T17:09:48.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.0 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58428: /' 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58430"' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58430' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58435: /' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58436"' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58436' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58440: /' 2024-06-11T17:09:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.2 2024-06-11T17:09:48.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-06-11T17:09:48.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58444"' 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58444' 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58449: /' 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58450"' 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58450' 2024-06-11T17:09:48.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.3 2024-06-11T17:09:48.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.4 2024-06-11T17:09:48.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58460"' 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58460' 2024-06-11T17:09:48.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.823 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-06-11T17:09:48.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-06-11T17:09:48.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58464"' 2024-06-11T17:09:48.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58464' 2024-06-11T17:09:48.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58453: /' 2024-06-11T17:09:48.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58466"' 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58466' 2024-06-11T17:09:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.5 2024-06-11T17:09:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58463: /' 2024-06-11T17:09:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58472: /' 2024-06-11T17:09:48.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-06-11T17:09:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.7 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58475"' 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58475' 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58469: /' 2024-06-11T17:09:48.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.6 2024-06-11T17:09:48.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58485"' 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58485' 2024-06-11T17:09:48.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58484: /' 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.8 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58490"' 2024-06-11T17:09:48.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58490' 2024-06-11T17:09:48.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58488: /' 2024-06-11T17:09:48.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.9 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58496"' 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58496' 2024-06-11T17:09:48.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.b 2024-06-11T17:09:48.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58502"' 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58502' 2024-06-11T17:09:48.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.a 2024-06-11T17:09:48.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58500: /' 2024-06-11T17:09:48.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58495: /' 2024-06-11T17:09:48.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.c 2024-06-11T17:09:48.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58505: /' 2024-06-11T17:09:48.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58512"' 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58512' 2024-06-11T17:09:48.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-06-11T17:09:48.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-06-11T17:09:48.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58518"' 2024-06-11T17:09:48.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58518' 2024-06-11T17:09:48.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-06-11T17:09:48.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-06-11T17:09:48.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58522"' 2024-06-11T17:09:48.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58522' 2024-06-11T17:09:48.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.d 2024-06-11T17:09:48.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-06-11T17:09:48.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-06-11T17:09:48.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58529"' 2024-06-11T17:09:48.951 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58529' 2024-06-11T17:09:48.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58527: /' 2024-06-11T17:09:48.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58521: /' 2024-06-11T17:09:48.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58525: /' 2024-06-11T17:09:48.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.e 2024-06-11T17:09:48.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-06-11T17:09:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-06-11T17:09:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58534"' 2024-06-11T17:09:48.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58534' 2024-06-11T17:09:48.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.f 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58539"' 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58539' 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.10 2024-06-11T17:09:48.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58537: /' 2024-06-11T17:09:48.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58546: /' 2024-06-11T17:09:48.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.11 2024-06-11T17:09:48.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-06-11T17:09:48.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-06-11T17:09:48.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:48.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:48.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58551"' 2024-06-11T17:09:48.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58551' 2024-06-11T17:09:48.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58549: /' 2024-06-11T17:09:49.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-06-11T17:09:49.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-06-11T17:09:49.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.12 2024-06-11T17:09:49.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58557"' 2024-06-11T17:09:49.010 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58557' 2024-06-11T17:09:49.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58556: /' 2024-06-11T17:09:49.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.13 2024-06-11T17:09:49.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.14 2024-06-11T17:09:49.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58564: /' 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58569"' 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58569' 2024-06-11T17:09:49.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-06-11T17:09:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-06-11T17:09:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58577"' 2024-06-11T17:09:49.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58577' 2024-06-11T17:09:49.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58572: /' 2024-06-11T17:09:49.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.15 2024-06-11T17:09:49.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-06-11T17:09:49.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-06-11T17:09:49.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58584"' 2024-06-11T17:09:49.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58584' 2024-06-11T17:09:49.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-06-11T17:09:49.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-06-11T17:09:49.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58587"' 2024-06-11T17:09:49.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58587' 2024-06-11T17:09:49.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-06-11T17:09:49.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.16 2024-06-11T17:09:49.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-06-11T17:09:49.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58596"' 2024-06-11T17:09:49.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58596' 2024-06-11T17:09:49.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.17 2024-06-11T17:09:49.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58590: /' 2024-06-11T17:09:49.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58583: /' 2024-06-11T17:09:49.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58595: /' 2024-06-11T17:09:49.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.18 2024-06-11T17:09:49.112 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58604"' 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58604' 2024-06-11T17:09:49.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.19 2024-06-11T17:09:49.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-06-11T17:09:49.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-06-11T17:09:49.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58611"' 2024-06-11T17:09:49.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58611' 2024-06-11T17:09:49.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58603: /' 2024-06-11T17:09:49.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-06-11T17:09:49.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-06-11T17:09:49.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58618"' 2024-06-11T17:09:49.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58618' 2024-06-11T17:09:49.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58623"' 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58623' 2024-06-11T17:09:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58621: /' 2024-06-11T17:09:49.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58627: /' 2024-06-11T17:09:49.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1a 2024-06-11T17:09:49.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58624: /' 2024-06-11T17:09:49.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1b 2024-06-11T17:09:49.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1d 2024-06-11T17:09:49.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58633: /' 2024-06-11T17:09:49.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-06-11T17:09:49.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-06-11T17:09:49.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58642"' 2024-06-11T17:09:49.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58642' 2024-06-11T17:09:49.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-11T17:09:49.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1c 2024-06-11T17:09:49.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-06-11T17:09:49.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-06-11T17:09:49.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 58655"' 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 58655' 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 58425 58430 58436 58444 58450 58460 58464 58466 58475 58485 58490 58496 58502 58512 58518 58522 58529 58534 58539 58551 58557 58569 58577 58584 58587 58596 58604 58611 58618 58623 58642 58655' 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:49.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58425 2024-06-11T17:09:49.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58651: /' 2024-06-11T17:09:49.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1e 2024-06-11T17:09:49.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: pg_scrub_mod 1.1f 2024-06-11T17:09:49.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/58667: /' 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: instructing pg 1.7 on osd.6 to scrub 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: dumped pgs 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:56.965 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.069 INFO:tasks.workunit.client.0.smithi012.stderr:58472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-06-11T17:09:57.069 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:57.070 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: instructing pg 1.1b on osd.0 to scrub 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: dumped pgs 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stat58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-06-11T17:09:57.145 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: instructing pg 1.4 on osd.3 to scrub 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: dumped pgs 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-06-11T17:09:57.146 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:58453: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:57.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-06-11T17:09:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: instructing pg 1.2 on osd.0 to scrub 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: dumped pgs 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:57.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.138 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-06-11T17:09:58.138 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-06-11T17:09:58.138 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: instructing pg 1.1 on osd.5 to scrub 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: dumped pgs 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:58435: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-11T17:09:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:58435: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.336 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: instructing pg 1.6 on osd.6 to scrub 2024-06-11T17:09:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: dumped pgs 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-06-11T17:09:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:58469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.348 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-06-11T17:09:58.348 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-06-11T17:09:58.348 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: instructing pg 1.9 on osd.1 to scrub 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.349 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: dumped pgs 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.350 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.351 INFO:tasks.workunit.client.0.smithi012.stderr:58488: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-11T17:09:58.351 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.351 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.351 INFO:tasks.workunit.client.0.smithi012.stderr:58488: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.416 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-06-11T17:09:58.416 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-06-11T17:09:58.416 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-11T17:09:58.416 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.416 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: instructing pg 1.5 on osd.4 to scrub 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-06-11T17:09:58.417 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: dumped pgs 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-11T17:09:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.419 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.419 INFO:tasks.workunit.client.0.smithi012.stderr:58463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-11T17:09:58.419 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.419 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.419 INFO:tasks.workunit.client.0.smithi012.stderr:58463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-11T17:09:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: instructing pg 1.8 on osd.5 to scrub 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: dumped pgs 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-11T17:09:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.455 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.455 INFO:tasks.workunit.client.0.smithi012.stderr:58484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.473 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: instructing pg 1.1c on osd.0 to scrub 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: dumped pgs 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.474 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.475 INFO:tasks.workunit.client.0.smithi012.stderr:58627: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.545 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: instructing pg 1.a on osd.1 to scrub 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: dumped pgs 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.546 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:58495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.609 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: instructing pg 1.16 on osd.6 to scrub 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: dumped pgs 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-06-11T17:09:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:58583: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: instructing pg 1.e on osd.1 to scrub 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: dumped pgs 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:58525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: instructing pg 1.0 on osd.7 to scrub 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:58428: dumped pgs 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:58428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58430 2024-06-11T17:09:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58436 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58444 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.685 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: instructing pg 1.f on osd.6 to scrub 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: dumped pgs 2024-06-11T17:09:58.686 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.687 INFO:tasks.workunit.client.0.smithi012.stderr:58527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.724 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.725 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: instructing pg 1.c on osd.3 to scrub 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: dumped pgs 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.726 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:09:54.013094+0000 '>' '' 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:58505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: instructing pg 1.3 on osd.4 to scrub 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: dumped pgs 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:58449: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58450 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58460 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58464 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58466 2024-06-11T17:09:58.739 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-06-11T17:09:58.740 INFO:tasks.workunit.client.0.smithi012.stderr:58556: instructing pg 1.13 on osd.4 to scrub 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: dumped pgs 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-06-11T17:09:58.741 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:58556: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.789 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.790 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: instructing pg 1.1d on osd.1 to scrub 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: dumped pgs 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.804 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: instructing pg 1.12 on osd.0 to scrub 2024-06-11T17:09:58.805 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: dumped pgs 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-06-11T17:09:58.806 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:58549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.830 INFO:tasks.workunit.client.0.smithi012.stderr:lect(.pgid=="1.7") | .last_scrub_stamp' 2024-06-11T17:09:58.830 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:58472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 2 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-11T17:09:58.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-11T17:09:58.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58475 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58485 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58490 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:58.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58496 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:s | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:58621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-11T17:09:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 2 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-06-11T17:09:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:58564: instructing pg 1.14 on osd.2 to scrub 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: dumped pgs 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:09:55.058593+0000 '>' '' 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:58564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.916 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-06-11T17:09:58.916 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-06-11T17:09:58.916 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-11T17:09:58.916 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: instructing pg 1.10 on osd.5 to scrub 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.917 INFO:tasks.workunit.client.0.smithi012.stderr:58537: dumped pgs 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:58537: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-11T17:09:58.919 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.919 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.919 INFO:tasks.workunit.client.0.smithi012.stderr:58537: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.926 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: instructing pg 1.11 on osd.0 to scrub 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: dumped pgs 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.927 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.928 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-11T17:09:58.929 INFO:tasks.workunit.client.0.smithi012.stderr:58546: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.929 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.929 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.929 INFO:tasks.workunit.client.0.smithi012.stderr:58546: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:58.983 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:58.984 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: instructing pg 1.19 on osd.3 to scrub 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: dumped pgs 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:58.985 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:58603: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:58.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.027 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: instructing pg 1.1a on osd.1 to scrub 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: dumped pgs 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.028 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:58624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-11T17:09:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:58624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-11T17:09:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: instructing pg 1.1e on osd.3 to scrub 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-06-11T17:09:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:58651: dumped pgs 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-11T17:09:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:58651: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.058 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-06-11T17:09:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:58521: instructing pg 1.d on osd.7 to scrub 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: dumped pgs 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:58521: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.151 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: instructing pg 1.b on osd.1 to scrub 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: dumped pgs 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:58500: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58502 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58512 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58518 2024-06-11T17:09:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58522 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58529 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58534 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58539 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58551 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58557 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58569 2024-06-11T17:09:59.167 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-06-11T17:09:59.167 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-06-11T17:09:59.167 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-11T17:09:59.167 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.167 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: instructing pg 1.18 on osd.5 to scrub 2024-06-11T17:09:59.168 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: dumped pgs 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-11T17:09:59.169 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:58595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:58595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.195 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: instructing pg 1.17 on osd.0 to scrub 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: dumped pgs 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:58590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.199 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: instructing pg 1.1f on osd.5 to scrub 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: dumped pgs 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.200 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.201 INFO:tasks.workunit.client.0.smithi012.stderr:58667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-11T17:09:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: instructing pg 1.15 on osd.1 to scrub 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: dumped pgs 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-11T17:08:16.654881+0000 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-11T17:09:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-11T17:08:16.654881+0000 '>' '' 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:58572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58577 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58584 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58587 2024-06-11T17:09:59.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58596 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58604 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58611 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58618 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58623 2024-06-11T17:09:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58642 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58655 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-11T17:09:59.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.2.pid 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.3.pid 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.7.pid 2024-06-11T17:09:59.251 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.4.pid 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/mon.a.pid 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.1.pid 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.0.pid 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53457 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:49329 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:50705 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:45633 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:56209 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:52081 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:45549 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:47953 2024-06-11T17:09:59.252 INFO:tasks.workunit.client.0.smithi012.stderr:46577 2024-06-11T17:09:59.253 INFO:tasks.workunit.client.0.smithi012.stderr:54833' 2024-06-11T17:09:59.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53457 49329 50705 45633 56209 52081 45549 47953 46577 54833 2024-06-11T17:09:59.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-06-11T17:09:59.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:09:59.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:09:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:09:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:09:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:10:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:10:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-06-11T17:10:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-06-11T17:10:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-06-11T17:10:04.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-06-11T17:10:04.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-06-11T17:10:04.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-06-11T17:10:04.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-06-11T17:10:04.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-06-11T17:10:04.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-06-11T17:10:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-06-11T17:10:04.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-06-11T17:10:04.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-06-11T17:10:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-06-11T17:10:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-06-11T17:10:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-06-11T17:10:04.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-06-11T17:10:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-06-11T17:10:04.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-06-11T17:10:04.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-06-11T17:10:04.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-06-11T17:10:04.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-06-11T17:10:04.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-06-11T17:10:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-06-11T17:10:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T17:10:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-06-11T17:10:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-06-11T17:10:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-06-11T17:10:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-11T17:10:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:10:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-11T17:10:04.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:10:04.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:10:04.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:10:04.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:10:04.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:10:04.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:10:04.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:10:04.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:10:04.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:10:04.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:10:04.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:10:04.658 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:10:04.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:10:04.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:10:04.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:10:04.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:10:04.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-11T17:10:04.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:10:04.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:04.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:10:04.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36888 2024-06-11T17:10:04.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:10:04.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:10:04.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-06-11T17:10:04.690 INFO:tasks.workunit.client.0.smithi012.stdout:TEST PASSED 2024-06-11T17:10:04.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-06-11T17:10:04.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-06-11T17:10:04.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-scrub 0 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:10:04.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:10:04.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:10:04.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:10:04.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:10:04.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:10:04.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:10:04.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:10:04.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:10:04.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:10:04.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:10:04.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:10:04.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-11T17:10:04.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-11T17:10:04.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:10:04.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:04.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36888 2024-06-11T17:10:04.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36888 2024-06-11T17:10:04.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:10:04.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:10:04.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-11T17:10:04.701 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-11T17:10:04.701 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-11T17:10:04.756 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-06-11T17:10:04.757 DEBUG:teuthology.orchestra.run.smithi012:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-06-11T17:10:04.812 INFO:tasks.workunit.client.0.smithi012.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-11T17:10:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-11T17:10:04.819 INFO:tasks.workunit.client.0.smithi012.stdout:This test is disabled 2024-06-11T17:10:04.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-11T17:10:04.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-11T17:10:04.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-scrub-dump 2024-06-11T17:10:04.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-06-11T17:10:04.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-scrub-dump 0 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:10:04.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:10:04.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:10:04.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:10:04.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:10:04.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:10:04.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:10:04.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:10:04.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:10:04.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:10:04.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:10:04.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:10:04.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-11T17:10:04.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-06-11T17:10:04.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:10:04.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:04.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61880 2024-06-11T17:10:04.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61880 2024-06-11T17:10:04.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:10:04.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:10:04.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-11T17:10:04.833 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-11T17:10:04.833 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-11T17:10:04.887 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-06-11T17:10:04.887 DEBUG:teuthology.orchestra.run.smithi012:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:++ TIMEOUT=300 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:++ MAX_TIMEOUT=15 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:++ PG_NUM=4 2024-06-11T17:10:04.941 INFO:tasks.workunit.client.0.smithi012.stderr:++ TMPDIR=/tmp 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ type xmlstarlet 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ XMLSTARLET=xmlstarlet 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:+++ uname 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' Linux = FreeBSD ']' 2024-06-11T17:10:04.942 INFO:tasks.workunit.client.0.smithi012.stderr:++ SED=sed 2024-06-11T17:10:04.943 INFO:tasks.workunit.client.0.smithi012.stderr:++ AWK=awk 2024-06-11T17:10:04.943 INFO:tasks.workunit.client.0.smithi012.stderr:+++ stty -a 2024-06-11T17:10:04.943 INFO:tasks.workunit.client.0.smithi012.stderr:+++ head -1 2024-06-11T17:10:04.943 INFO:tasks.workunit.client.0.smithi012.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-06-11T17:10:04.944 INFO:tasks.workunit.client.0.smithi012.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-11T17:10:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:++ termwidth= 2024-06-11T17:10:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-06-11T17:10:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:++ DIFFCOLOPTS='-y ' 2024-06-11T17:10:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:++ KERNCORE=kernel.core_pattern 2024-06-11T17:10:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:++ EXTRA_OPTS= 2024-06-11T17:10:04.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ test '' = TESTS 2024-06-11T17:10:04.948 INFO:tasks.workunit.client.0.smithi012.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-06-11T17:10:04.949 INFO:tasks.workunit.client.0.smithi012.stderr:++ uname 2024-06-11T17:10:04.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' Linux = FreeBSD ']' 2024-06-11T17:10:04.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ use_ec_overwrite=true 2024-06-11T17:10:04.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ getjson=no 2024-06-11T17:10:04.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ jqfilter='def walk(f): 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T17:10:04.951 INFO:tasks.workunit.client.0.smithi012.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ main osd-scrub-repair 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ local dir=td/osd-scrub-repair 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ shift 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ shopt -s -o xtrace 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-11T17:10:04.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:10:04.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-11T17:10:04.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-11T17:10:04.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-11T17:10:04.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-11T17:10:04.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-scrub-repair 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-06-11T17:10:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none ' 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-06-11T17:10:04.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-11T17:10:04.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-06-11T17:10:04.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-11T17:10:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-06-11T17:10:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_bluestore_basic 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_bluestore_failed 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_bluestore_scrub 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_bluestore_tag 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_erasure_coded_appends 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_and_repair_lrc_appends 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_and_repair_replicated 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_scrub_erasure_appends 2024-06-11T17:10:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_scrub_replicated 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_corrupt_snapset_scrub_rep 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_list_missing_erasure_coded_appends 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_list_missing_erasure_coded_overwrites 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_periodic_scrub_replicated 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_repair_stats 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_repair_stats_ec 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_request_scrub_priority 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_scrub_warning 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_skip_non_repair_during_recovery 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_unfound_erasure_coded_appends 2024-06-11T17:10:04.964 INFO:tasks.workunit.client.0.smithi012.stderr:TEST_unfound_erasure_coded_overwrites' 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:10:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:10:04.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:10:04.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:10:04.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:10:04.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:10:04.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:10:04.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:10:04.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:10:04.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:10:04.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:10:04.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:10:04.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:10:04.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:10:04.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:10:04.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:10:04.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:10:04.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:10:04.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:10:04.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:10:04.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:10:04.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:04.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:10:04.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:10:04.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:10:04.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:10:04.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:10:04.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:04.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:04.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:10:04.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:10:04.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-11T17:10:04.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-11T17:10:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:10:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:10:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-06-11T17:10:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-06-11T17:10:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-06-11T17:10:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-11T17:10:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:10:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:10:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:10:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:10:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:10:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-11T17:10:05.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:10:05.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:10:05.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:10:05.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:10:05.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:05.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:05.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:05.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-11T17:10:05.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:10:05.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:10:05.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:10:05.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:10:05.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:10:05.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:10:05.700 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:10:05.700 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:10:05.700 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:10:05.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:10:05.819 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:10:05.819 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:10:05.819 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:10:05.819 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:10:05.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:10:05.820 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:10:05.820 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:10:05.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:10:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:10:06.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:10:06.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:10:06.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:10:06.825 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:10:06.825 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:06.825 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:06.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:06.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:10:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:10:06.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-11T17:10:06.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:10:06.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:10:06.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:10:06.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:10:06.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:10:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:10:06.870 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:10:06.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1dc3c275-4c2f-4665-8209-4cc486a230dd 2024-06-11T17:10:06.871 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 1dc3c275-4c2f-4665-8209-4cc486a230dd 2024-06-11T17:10:06.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1dc3c275-4c2f-4665-8209-4cc486a230dd' 2024-06-11T17:10:06.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:10:06.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuhGhmTon7NBAAr05SF+I4NoW9Ce7hTs+PQw== 2024-06-11T17:10:06.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuhGhmTon7NBAAr05SF+I4NoW9Ce7hTs+PQw=="}' 2024-06-11T17:10:06.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1dc3c275-4c2f-4665-8209-4cc486a230dd -i td/osd-scrub-repair/0/new.json 2024-06-11T17:10:07.142 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:10:07.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:10:07.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDuhGhmTon7NBAAr05SF+I4NoW9Ce7hTs+PQw== --osd-uuid 1dc3c275-4c2f-4665-8209-4cc486a230dd 2024-06-11T17:10:07.201 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.201+0000 7fa3e6ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:07.208 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.209+0000 7fa3e6ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:07.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.209+0000 7fa3e6ee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.209+0000 7fa3e6ee37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:10:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.209+0000 7fa3e6ee37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:10:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.209+0000 7fa3e6ee37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:10:07.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:07.301+0000 7fa3e6ee37c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:10:14.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:10:14.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:10:14.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:10:14.032 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:10:14.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:10:14.446 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:10:14.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:10:14.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-11T17:10:14.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:10:14.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:10:14.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:10:14.482 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:14.481+0000 7fe779c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:14.506 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:14.505+0000 7fe779c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:14.508 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:14.509+0000 7fe779c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:14.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:10:14.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:10:14.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:10:14.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:10:14.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:10:14.801 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:10:14.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:14.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:10:14.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:14.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:15.053+0000 7fe779c767c0 -1 Falling back to public interface 2024-06-11T17:10:15.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:16.151 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:10:16.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:10:16.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:16.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:16.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:17.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:17.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:17.490 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:10:17.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:10:17.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:17.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:17.809+0000 7fe779c767c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:10:17.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:18.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:18.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:18.821 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:10:18.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:10:18.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:18.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:19.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:20.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:20.186 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:10:20.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:10:20.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:20.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:20.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:21.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:21.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:21.584 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:10:21.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:10:21.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:21.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:21.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:22.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:22.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:22.926 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:10:22.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:10:22.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:22.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:23.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:24.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:24.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:24.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:10:24.262 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:10:24.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:24.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:24.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:25.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:25.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:25.752 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:10:25.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:10:25.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:25.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:26.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:27.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:27.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:27.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:10:27.089 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:10:27.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:27.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:27.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:28.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:28.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:28.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:10:28.425 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:10:28.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:28.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:28.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:29.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:29.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:29.783 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:10:29.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:10:29.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:29.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:30.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:31.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:31.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:31.121 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:10:31.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:10:31.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:31.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:31.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:32.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:32.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:32.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:10:32.459 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:10:32.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:32.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:33.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:34.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:34.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:34.189 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:10:34.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:10:34.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:34.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:34.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:35.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:35.908 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:10:35.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:35.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:10:35.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:35.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:36.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:37.677 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:10:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:10:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:38.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:39.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:39.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:39.604 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:10:39.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:10:39.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:39.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:40.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:41.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:41.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:41.186 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:10:41.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:10:41.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:41.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:42.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:42.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:42.975 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:10:42.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:10:42.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:42.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:10:43.331 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4288391752,v1:127.0.0.1:6803/4288391752] [v2:127.0.0.1:6804/4288391752,v1:127.0.0.1:6805/4288391752] exists,up 1dc3c275-4c2f-4665-8209-4cc486a230dd 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:10:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:10:43.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:10:43.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:10:43.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:10:43.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:10:43.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:10:43.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:10:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c62959de-79f8-487d-99cf-2c6536ddf53e 2024-06-11T17:10:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c62959de-79f8-487d-99cf-2c6536ddf53e' 2024-06-11T17:10:43.339 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 c62959de-79f8-487d-99cf-2c6536ddf53e 2024-06-11T17:10:43.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:10:43.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAThWhmoEGaFRAA+b+EekTaWngwAR2MFHLTRQ== 2024-06-11T17:10:43.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAThWhmoEGaFRAA+b+EekTaWngwAR2MFHLTRQ=="}' 2024-06-11T17:10:43.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c62959de-79f8-487d-99cf-2c6536ddf53e -i td/osd-scrub-repair/1/new.json 2024-06-11T17:10:43.755 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:10:43.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:10:43.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAThWhmoEGaFRAA+b+EekTaWngwAR2MFHLTRQ== --osd-uuid c62959de-79f8-487d-99cf-2c6536ddf53e 2024-06-11T17:10:43.815 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.813+0000 7fa50fe4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:43.856 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.857+0000 7fa50fe4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:43.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.857+0000 7fa50fe4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:43.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.857+0000 7fa50fe4f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:10:43.860 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.857+0000 7fa50fe4f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:10:43.860 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.861+0000 7fa50fe4f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:10:43.925 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:43.925+0000 7fa50fe4f7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:10:50.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:10:50.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:10:50.802 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:10:50.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:10:50.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:10:51.219 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:10:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:10:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-11T17:10:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:10:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:10:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:10:51.255 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:51.253+0000 7f3314b477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:51.288 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:51.289+0000 7f3314b477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:51.290 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:51.289+0000 7f3314b477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:10:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:10:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:10:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:10:51.572 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:51.838 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:51.837+0000 7f3314b477c0 -1 Falling back to public interface 2024-06-11T17:10:51.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:52.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:52.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:52.926 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:10:52.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:10:52.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:52.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:53.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:54.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:54.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:54.256 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:10:54.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:10:54.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:54.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:54.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:54.627 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:10:54.625+0000 7f3314b477c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:10:55.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:55.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:55.592 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:10:55.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:10:55.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:55.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:56.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:57.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:57.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:57.113 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:10:57.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:10:57.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:57.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:58.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:58.453 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:10:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:10:58.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:58.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:10:58.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:10:59.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:10:59.871 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:10:59.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:10:59.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:10:59.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:10:59.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:00.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:01.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:01.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:01.304 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:11:01.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:11:01.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:01.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:01.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:02.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:02.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:02.713 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:11:02.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:11:02.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:02.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:03.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:04.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:04.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:04.047 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:11:04.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:11:04.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:04.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:04.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:05.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:05.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:05.553 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:11:05.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:11:05.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:05.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:05.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:06.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:06.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:06.960 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:11:06.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:11:06.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:06.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:07.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:08.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:08.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:08.308 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:11:08.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:11:08.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:08.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:08.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:09.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:09.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:09.666 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:11:09.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:11:09.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:09.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:10.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:11.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:11.286 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:11:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:11:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:12.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:13.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:13.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:13.082 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:11:13.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:11:13.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:13.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:13.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:14.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:14.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:14.591 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:11:14.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:11:14.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:14.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:15.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:16.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:16.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:16.027 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:11:16.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:11:16.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:16.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:16.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:17.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:17.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:17.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:11:17.908 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:11:17.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:17.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:18.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:19.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:19.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:19.360 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:11:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:11:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:19.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:20.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:20.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:20.863 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:11:20.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:11:20.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:20.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:21.235 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/512599800,v1:127.0.0.1:6811/512599800] [v2:127.0.0.1:6812/512599800,v1:127.0.0.1:6813/512599800] exists,up c62959de-79f8-487d-99cf-2c6536ddf53e 2024-06-11T17:11:21.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:11:21.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:11:21.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:11:21.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-06-11T17:11:21.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:11:21.597 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:11:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:11:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:11:22.053 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:11:22.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:11:23.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:11:25.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-06-11T17:11:25.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:11:25.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:11:25.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:11:25.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:11:25.445 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:11:25.446 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:11:25.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:11:25.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:11:25.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:11:25.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:11:25.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:11:25.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:11:25.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:11:25.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:11:25.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:11:25.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:11:25.907 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T17:11:25.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:11:25.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:25.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:11:26.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-06-11T17:11:26.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-06-11T17:11:26.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-06-11T17:11:26.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:26.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:11:26.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672968 2024-06-11T17:11:26.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672968 2024-06-11T17:11:26.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-06-11T17:11:26.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:11:26.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-06-11T17:11:26.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:11:26.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:11:26.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-06-11T17:11:26.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:11:26.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-06-11T17:11:26.239 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836492 2024-06-11T17:11:26.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-11T17:11:26.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:26.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836492 2024-06-11T17:11:26.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:11:27.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:11:27.562 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:27.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836492 2024-06-11T17:11:27.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:11:28.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T17:11:28.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:29.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-11T17:11:29.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:11:29.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:11:29.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672968 2024-06-11T17:11:29.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:11:29.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672968 2024-06-11T17:11:29.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:11:29.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672968 2024-06-11T17:11:29.248 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672968 2024-06-11T17:11:29.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-06-11T17:11:29.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:11:29.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672968 2024-06-11T17:11:29.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:11:29.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:11:29.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:11:30.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:11:30.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:11:30.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:11:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:11:30.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:11:30.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:11:30.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T17:11:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-11T17:11:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T17:11:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T17:11:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T17:11:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T17:11:31.308 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:11:31.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T17:11:31.953 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:11:31.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T17:11:31.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T17:11:31.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:11:32.378 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-06-11T17:11:32.379 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-11T17:11:32.379 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:11:32.379 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-11T17:11:32.379 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-11T17:11:32.380 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:11:32.380 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-11T17:11:32.380 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:11:32.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:11:32.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:11:32.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-11T17:11:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-11T17:11:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:11:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:11:33.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:11:33.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:11:33.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:11:33.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:11:33.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:11:33.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-11T17:11:35.025 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T17:11:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T17:11:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:11:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:11:35.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:11:35.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:11:35.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:11:35.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:11:35.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:11:35.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:11:35.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:11:35.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:11:35.604 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:35.605+0000 7f4dd93d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:35.606 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:35.605+0000 7f4dd93d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:35.608 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:35.609+0000 7f4dd93d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:35.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:11:36.149 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:36.149+0000 7f4dd93d97c0 -1 Falling back to public interface 2024-06-11T17:11:36.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:37.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:37.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:37.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:11:37.267 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:11:37.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:37.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:11:37.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:38.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:38.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:38.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:11:38.610 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:11:38.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:38.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:11:38.682 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:38.681+0000 7f4dd93d97c0 -1 osd.0 22 log_to_monitors true 2024-06-11T17:11:39.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:40.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:40.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:40.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:11:40.004 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:11:40.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:40.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:11:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/1217185006,v1:127.0.0.1:6803/1217185006] [v2:127.0.0.1:6804/1217185006,v1:127.0.0.1:6805/1217185006] exists,up 1dc3c275-4c2f-4665-8209-4cc486a230dd 2024-06-11T17:11:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:11:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:11:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:11:40.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:11:40.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:11:40.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:11:40.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:11:40.350 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:11:40.351 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:11:40.351 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:11:40.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:11:40.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:11:40.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:11:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:11:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:11:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:11:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:11:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:11:40.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:11:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:11:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T17:11:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:11:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:40.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:11:40.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=111669149698 2024-06-11T17:11:40.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 111669149698 2024-06-11T17:11:40.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-111669149698' 2024-06-11T17:11:40.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:40.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:11:41.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672972 2024-06-11T17:11:41.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672972 2024-06-11T17:11:41.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-111669149698 1-42949672972' 2024-06-11T17:11:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:11:41.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-111669149698 2024-06-11T17:11:41.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:11:41.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:11:41.123 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-111669149698 2024-06-11T17:11:41.123 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:11:41.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=111669149698 2024-06-11T17:11:41.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-06-11T17:11:41.124 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 111669149698 2024-06-11T17:11:41.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:41.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 111669149698 2024-06-11T17:11:41.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:11:42.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:11:42.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 111669149698 -lt 111669149698 2024-06-11T17:11:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:11:42.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672972 2024-06-11T17:11:42.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:11:42.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:11:42.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672972 2024-06-11T17:11:42.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:11:42.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672972 2024-06-11T17:11:42.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-11T17:11:42.790 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672972 2024-06-11T17:11:42.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:11:43.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672972 2024-06-11T17:11:43.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:11:43.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:11:43.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:11:43.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:11:43.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:11:43.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:11:43.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:11:43.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:11:43.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:11:43.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:11:43.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:11:43.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:11:43.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:11:43.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:11:43.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:11:44.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:11:44.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:11:44.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:11:44.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-11T17:11:44.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-11T17:11:44.342 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:11:44.342 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-11T17:11:44.342 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:11:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-11T17:11:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-11T17:11:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-06-11T17:11:44.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-06-11T17:11:44.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:44.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:44.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:44.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:45.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:45.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-06-11T17:11:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-11T17:11:45.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.3 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:45.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:45.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:45.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:45.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:45.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:21.980731+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:45.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:11:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:11:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:46.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:46.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:47.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:21.980731+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:47.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:11:48.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:11:48.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:48.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:48.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:48.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:48.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:48.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:48.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:21.980731+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:48.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:11:49.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:11:49.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:49.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:49.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:49.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:49.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:49.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:49.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:21.980731+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:49.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:11:50.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:11:50.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:50.688 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:50.688 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:50.688 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:50.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:50.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:51.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:21.980731+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:51.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:11:52.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:11:52.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:11:52.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T17:11:52.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T17:11:52.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:11:52.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:11:52.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T17:11:52.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:11:46.339220+0000 '>' 2024-06-11T17:11:21.980731+0000 2024-06-11T17:11:52.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:11:52.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:11:52.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:11:52.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:11:52.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:11:52.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:11:52.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:11:52.900 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T17:11:52.900 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:11:52.900 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:11:53.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:11:53.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:11:53.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:11:53.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:11:53.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:11:53.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:11:53.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:11:53.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:11:53.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:11:53.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:11:53.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:11:53.214 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:11:53.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:11:53.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:11:53.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:11:53.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:11:53.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:11:53.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:11:53.249 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:53.248+0000 7f6f5ba757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:53.252 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:53.252+0000 7f6f5ba757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:53.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:53.252+0000 7f6f5ba757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:11:53.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:53.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:53.798 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:53.796+0000 7f6f5ba757c0 -1 Falling back to public interface 2024-06-11T17:11:53.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:54.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:54.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:54.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:11:54.906 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:11:54.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:54.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:55.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:56.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:56.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:11:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:11:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:56.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:11:57.008 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:11:57.008+0000 7f6f5ba757c0 -1 osd.1 28 log_to_monitors true 2024-06-11T17:11:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:11:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:11:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:11:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:11:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:11:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:11:57.913 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/352622051,v1:127.0.0.1:6811/352622051] [v2:127.0.0.1:6812/352622051,v1:127.0.0.1:6813/352622051] exists,up c62959de-79f8-487d-99cf-2c6536ddf53e 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:11:57.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:11:57.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:11:58.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:11:58.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:11:58.470 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T17:11:58.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:11:58.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:58.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:11:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=111669149702 2024-06-11T17:11:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 111669149702 2024-06-11T17:11:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-111669149702' 2024-06-11T17:11:58.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:11:58.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:11:58.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953474 2024-06-11T17:11:58.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953474 2024-06-11T17:11:58.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-06-11T17:11:58.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:11:58.771 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-111669149702 2024-06-11T17:11:58.771 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:11:58.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:11:58.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-111669149702 2024-06-11T17:11:58.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:11:58.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=111669149702 2024-06-11T17:11:58.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-06-11T17:11:58.774 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 111669149702 2024-06-11T17:11:58.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:11:59.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 111669149701 -lt 111669149702 2024-06-11T17:11:59.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:12:00.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:12:00.095 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:12:00.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 111669149703 -lt 111669149702 2024-06-11T17:12:00.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:12:00.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953474 2024-06-11T17:12:00.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:12:00.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:12:00.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953474 2024-06-11T17:12:00.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:12:00.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953474 2024-06-11T17:12:00.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-06-11T17:12:00.434 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 137438953474 2024-06-11T17:12:00.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:12:00.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953474 -lt 137438953474 2024-06-11T17:12:00.771 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:12:00.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:12:00.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:12:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:12:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:12:01.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:12:01.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:12:01.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:12:01.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:12:01.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:12:01.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:12:01.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:12:01.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:12:01.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:12:01.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:12:01.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:12:01.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:12:01.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:12:01.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:12:01.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:12:02.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:12:02.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:12:02.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:12:02.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:12:02.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:12:02.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:12:02.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:12:02.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:12:02.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:12:02.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:12:02.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:12:02.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:12:02.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:12:02.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:12:02.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:12:02.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:12:02.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:12:02.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:12:02.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:12:02.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:12:02.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:12:02.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:12:02.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:12:02.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:12:02.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:12:02.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:12:02.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:12:02.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:12:02.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:12:02.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:12:02.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:12:02.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:12:02.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:12:02.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:12:02.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:12:02.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:12:02.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:12:02.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:12:02.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:12:02.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:12:02.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:12:02.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:12:02.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:12:02.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:12:02.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:12:02.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:12:02.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:12:02.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:12:02.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:12:02.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:12:02.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:12:02.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:12:02.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-11T17:12:02.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-11T17:12:02.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-11T17:12:04.004 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 edc62673-a932-407d-a4f6-ec46e7b063f8 2024-06-11T17:12:04.270 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:12:04.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.328+0000 7ff7809eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:04.336 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.336+0000 7ff7809eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:04.337 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.336+0000 7ff7809eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:04.338 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.336+0000 7ff7809eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:12:04.338 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.336+0000 7ff7809eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:12:04.338 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.336+0000 7ff7809eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:12:04.429 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:04.428+0000 7ff7809eb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:12:11.174 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:12:11.588 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:12:11.630 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:11.628+0000 7f450d6967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:11.644 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:11.644+0000 7f450d6967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:11.645 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:11.644+0000 7f450d6967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:11.932 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:12:12.201 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:12.200+0000 7f450d6967c0 -1 Falling back to public interface 2024-06-11T17:12:13.282 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:12:14.616 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:12:14.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:14.808+0000 7f450d6967c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:12:15.948 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:12:17.276 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:12:18.729 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:12:20.068 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:12:21.411 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:12:22.925 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:12:24.262 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:12:25.691 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:12:27.029 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:12:28.366 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:12:30.119 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:12:31.841 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:12:33.335 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:12:34.829 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:12:36.297 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:12:37.968 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:12:39.598 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:12:40.992 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:12:41.327 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3065174527,v1:127.0.0.1:6803/3065174527] [v2:127.0.0.1:6804/3065174527,v1:127.0.0.1:6805/3065174527] exists,up edc62673-a932-407d-a4f6-ec46e7b063f8 2024-06-11T17:12:41.332 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 e4ff3ba1-9e08-4aa3-8670-53f3436e2cf1 2024-06-11T17:12:41.741 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:12:41.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.804+0000 7fa3588987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:41.852 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.852+0000 7fa3588987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:41.854 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.852+0000 7fa3588987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:41.855 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.852+0000 7fa3588987c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:12:41.855 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.852+0000 7fa3588987c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:12:41.855 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.852+0000 7fa3588987c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:12:41.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:41.932+0000 7fa3588987c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:12:48.700 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:12:49.111 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:12:49.149 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:49.148+0000 7f51fca147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:49.152 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:49.152+0000 7f51fca147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:49.153 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:49.152+0000 7f51fca147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:12:49.436 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:12:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:49.700+0000 7f51fca147c0 -1 Falling back to public interface 2024-06-11T17:12:50.795 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:12:52.124 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:12:52.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:12:52.388+0000 7f51fca147c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:12:53.458 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:12:54.795 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:12:56.185 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:12:57.527 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:12:58.860 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:13:00.240 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:13:01.571 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:13:02.901 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:13:04.230 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:13:05.567 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:13:06.920 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:13:08.646 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:13:10.105 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:13:11.836 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:13:13.564 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:13:15.363 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:13:16.825 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:13:18.221 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:13:18.582 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1762747292,v1:127.0.0.1:6811/1762747292] [v2:127.0.0.1:6812/1762747292,v1:127.0.0.1:6813/1762747292] exists,up e4ff3ba1-9e08-4aa3-8670-53f3436e2cf1 2024-06-11T17:13:18.588 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 1a2d3fa1-6211-418c-a3e1-258561a00202 2024-06-11T17:13:19.023 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:13:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.088+0000 7faf12df07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:19.124 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.124+0000 7faf12df07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:19.126 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.124+0000 7faf12df07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:19.127 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.124+0000 7faf12df07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:13:19.127 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.124+0000 7faf12df07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:13:19.127 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.124+0000 7faf12df07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:13:19.212 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:19.212+0000 7faf12df07c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:13:26.279 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:13:26.700 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:13:26.741 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:26.740+0000 7fd47e2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:26.762 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:26.760+0000 7fd47e2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:26.764+0000 7fd47e2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:13:27.044 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:13:27.325 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:27.324+0000 7fd47e2847c0 -1 Falling back to public interface 2024-06-11T17:13:28.367 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:13:29.692 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:13:29.915 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:13:29.916+0000 7fd47e2847c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:13:31.046 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:13:32.398 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:13:33.732 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:13:35.059 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:13:36.469 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:13:37.911 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:13:39.248 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:13:40.670 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:13:42.083 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:13:43.420 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:13:44.906 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:13:46.956 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:13:48.324 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:13:49.945 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:13:51.524 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:13:53.437 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:13:55.000 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:13:56.448 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:13:56.826 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1787845779,v1:127.0.0.1:6819/1787845779] [v2:127.0.0.1:6820/1787845779,v1:127.0.0.1:6821/1787845779] exists,up 1a2d3fa1-6211-418c-a3e1-258561a00202 2024-06-11T17:13:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T17:13:59.626 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836501 2024-06-11T17:14:01.342 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672976 2024-06-11T17:14:01.698 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509447 2024-06-11T17:14:06.733 INFO:tasks.workunit.client.0.smithi012.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-06-11T17:14:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-06-11T17:14:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-06-11T17:14:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-06-11T17:14:07.176 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T17:14:07.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:14:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:14:07.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:14:07.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:14:07.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:14:07.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:14:07.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:14:07.773 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:07.773 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:14:07.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:14:07.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:07.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:14:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-06-11T17:14:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-06-11T17:14:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-06-11T17:14:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:07.922 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:14:08.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-06-11T17:14:08.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-06-11T17:14:08.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-06-11T17:14:08.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:08.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:14:08.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-06-11T17:14:08.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-06-11T17:14:08.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509450' 2024-06-11T17:14:08.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:08.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-06-11T17:14:08.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:08.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:14:08.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-06-11T17:14:08.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:08.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-06-11T17:14:08.214 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836504 2024-06-11T17:14:08.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-11T17:14:08.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:08.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836504 2024-06-11T17:14:08.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:14:09.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:14:09.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:09.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-11T17:14:09.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:09.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-06-11T17:14:09.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:09.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:14:09.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-06-11T17:14:09.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:09.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-06-11T17:14:09.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-11T17:14:09.873 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672978 2024-06-11T17:14:09.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:10.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-11T17:14:10.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:10.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-06-11T17:14:10.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:10.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:14:10.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-06-11T17:14:10.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:10.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-06-11T17:14:10.214 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509450 2024-06-11T17:14:10.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-11T17:14:10.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:10.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-11T17:14:10.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:14:10.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:10.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:10.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:14:10.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:10.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:10.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:10.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:10.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:10.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:10.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:14:11.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:11.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:11.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:11.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:14:11.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:14:11.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:14:11.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-06-11T17:14:11.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-06-11T17:14:11.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:14:11.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-11T17:14:11.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:14:11.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:14:11.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:14:11.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:14:11.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:14:11.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:11.846 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:14:12.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:14:12.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:12.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:14:12.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:14:12.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:14:12.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:14:12.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:14:12.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:13.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-11T17:14:15.284 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:15.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:14:15.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:14:15.822 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:14:15.822 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:14:15.822 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:14:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:14:15.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:14:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:14:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:14:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:14:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:14:15.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:15.855+0000 7f2c397cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:15.861 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:15.859+0000 7f2c397cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:15.863 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:15.859+0000 7f2c397cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:14:16.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:16.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:14:16.163 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:14:16.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:16.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:16.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:16.403+0000 7f2c397cf7c0 -1 Falling back to public interface 2024-06-11T17:14:16.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:17.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:17.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:17.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:14:17.511 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:17.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:17.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:17.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:18.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:18.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:18.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:14:18.846 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:14:18.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:18.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:19.247 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:19.247+0000 7f2c397cf7c0 -1 osd.0 21 log_to_monitors true 2024-06-11T17:14:19.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:20.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:20.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:20.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:14:20.277 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:14:20.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:20.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:20.613 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/432132666,v1:127.0.0.1:6803/432132666] [v2:127.0.0.1:6804/432132666,v1:127.0.0.1:6805/432132666] exists,up edc62673-a932-407d-a4f6-ec46e7b063f8 2024-06-11T17:14:20.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:14:20.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:14:20.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:14:20.615 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:14:20.615 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:14:20.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:14:20.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:14:20.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:14:20.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:14:20.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:14:20.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:14:20.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:14:20.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:14:20.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:14:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:14:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:14:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:14:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:21.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:14:21.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182402 2024-06-11T17:14:21.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182402 2024-06-11T17:14:21.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402' 2024-06-11T17:14:21.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:21.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:14:21.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-06-11T17:14:21.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-06-11T17:14:21.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672982' 2024-06-11T17:14:21.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:21.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:14:21.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-06-11T17:14:21.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-06-11T17:14:21.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672982 2-64424509453' 2024-06-11T17:14:21.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:21.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182402 2024-06-11T17:14:21.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:21.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:14:21.521 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182402 2024-06-11T17:14:21.521 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:21.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182402 2024-06-11T17:14:21.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-11T17:14:21.523 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 107374182402 2024-06-11T17:14:21.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:21.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 107374182402 2024-06-11T17:14:21.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:14:22.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:14:22.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-11T17:14:23.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:23.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:23.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-06-11T17:14:23.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:14:23.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-06-11T17:14:23.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:23.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-06-11T17:14:23.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-11T17:14:23.216 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672982 2024-06-11T17:14:23.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:23.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-06-11T17:14:23.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:23.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-06-11T17:14:23.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:23.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:14:23.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:23.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-06-11T17:14:23.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-06-11T17:14:23.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-11T17:14:23.553 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509453 2024-06-11T17:14:23.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:24.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509453 2024-06-11T17:14:24.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:14:24.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:24.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:24.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:14:24.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:24.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:24.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:14:24.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:24.814 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:24.814 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:25.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:14:25.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:14:25.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:14:25.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-11T17:14:25.387 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: { 2024-06-11T17:14:25.388 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-11T17:14:25.388 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:14:25.401 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: { 2024-06-11T17:14:25.401 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-11T17:14:25.401 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:14:25.414 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2: { 2024-06-11T17:14:25.415 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-11T17:14:25.415 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:14:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-06-11T17:14:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T17:14:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:14:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:14:25.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-06-11T17:14:25.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-06-11T17:14:25.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:14:25.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:14:25.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:25.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:14:26.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-06-11T17:14:26.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-06-11T17:14:26.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:26.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:26.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:26.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:26.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:26.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-06-11T17:14:26.587 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:14:26.587 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:14:26.587 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T17:14:26.588 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-05-28T17:12:46.590199+0000" 2024-06-11T17:14:26.588 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:14:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T17:14:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:26.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:26.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:26.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:26.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:13:57.287070+0000 '>' 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:26.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:14:27.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:14:27.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:14:27.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:14:27.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:27.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:27.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:27.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:13:57.287070+0000 '>' 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:28.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:14:29.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:14:29.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:14:29.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:14:29.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:29.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:29.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:29.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:29.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:13:57.287070+0000 '>' 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:29.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:14:30.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:14:30.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:14:30.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:14:30.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:30.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:30.574 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:30.574 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:30.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:13:57.287070+0000 '>' 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:31.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:14:31.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:14:31.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:14:31.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:14:31.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:14:31.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:14:31.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:14:31.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:14:27.304673+0000 '>' 2024-06-11T17:13:57.287070+0000 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:14:32.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:14:32.241 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:14:32.241 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:14:32.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:14:32.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:14:32.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:14:32.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:14:32.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:14:32.675 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:32.675 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:14:32.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:14:32.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:32.676 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:14:32.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182405 2024-06-11T17:14:32.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182405 2024-06-11T17:14:32.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405' 2024-06-11T17:14:32.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:32.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:14:32.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-11T17:14:32.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-11T17:14:32.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405 1-42949672985' 2024-06-11T17:14:32.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:32.995 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:14:33.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-06-11T17:14:33.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-06-11T17:14:33.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405 1-42949672985 2-64424509457' 2024-06-11T17:14:33.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:33.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182405 2024-06-11T17:14:33.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:33.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:14:33.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182405 2024-06-11T17:14:33.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:33.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182405 2024-06-11T17:14:33.143 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 107374182405 2024-06-11T17:14:33.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-06-11T17:14:33.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:33.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182405 -lt 107374182405 2024-06-11T17:14:33.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:33.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-11T17:14:33.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:33.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:14:33.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-11T17:14:33.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:33.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-11T17:14:33.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-11T17:14:33.450 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672985 2024-06-11T17:14:33.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:33.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672985 2024-06-11T17:14:33.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:14:34.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:14:34.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:35.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672985 2024-06-11T17:14:35.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:35.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-06-11T17:14:35.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:35.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:14:35.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-06-11T17:14:35.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:35.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-06-11T17:14:35.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-11T17:14:35.141 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509457 2024-06-11T17:14:35.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:35.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-11T17:14:35.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:14:35.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:35.489 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:35.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:36.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:14:36.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:36.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:36.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:36.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:14:36.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:14:36.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:14:36.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-06-11T17:14:36.968 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:14:36.968 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:14:27.549016+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-06-11T17:14:27.304673+0000 21'1 2024-06-11T17:14:27.304673+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:14:27.304673+0000 1 0 2024-06-11T17:14:36.968 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:14:36.968 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:14:36.969 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:14:36.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-11T17:14:36.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:14:36.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:14:36.987 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:14:36.988 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:14:36.988 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:14:36.988 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:36.988 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:14:37.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:14:37.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:14:37.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:14:37.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:14:37.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:14:37.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:37.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-11T17:14:38.111 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:14:38.111 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:14:38.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:14:38.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:14:38.416 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:14:38.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:14:38.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:14:38.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:14:38.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:14:38.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:14:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:38.451+0000 7f559f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:38.478 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:38.475+0000 7f559f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:38.479 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:38.479+0000 7f559f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:38.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:14:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:38.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:39.013 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:39.011+0000 7f559f7dc7c0 -1 Falling back to public interface 2024-06-11T17:14:39.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:14:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:40.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:40.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:41.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:14:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:14:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:41.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:42.109 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:42.107+0000 7f559f7dc7c0 -1 osd.0 26 log_to_monitors true 2024-06-11T17:14:42.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:42.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:42.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:14:42.802 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:14:42.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:42.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1022106811,v1:127.0.0.1:6803/1022106811] [v2:127.0.0.1:6804/1022106811,v1:127.0.0.1:6805/1022106811] exists,up edc62673-a932-407d-a4f6-ec46e7b063f8 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:14:43.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:14:43.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:14:43.261 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:14:43.261 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:14:43.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:14:43.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:14:43.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:14:43.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:14:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:14:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:14:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:14:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:14:43.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:43.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:14:43.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T17:14:43.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T17:14:43.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882' 2024-06-11T17:14:43.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:43.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:14:44.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-06-11T17:14:44.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-06-11T17:14:44.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672989' 2024-06-11T17:14:44.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:44.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:14:44.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-06-11T17:14:44.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-06-11T17:14:44.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672989 2-64424509460' 2024-06-11T17:14:44.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:44.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018882 2024-06-11T17:14:44.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:44.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:14:44.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018882 2024-06-11T17:14:44.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:44.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T17:14:44.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-11T17:14:44.242 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 128849018882 2024-06-11T17:14:44.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:44.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 128849018882 2024-06-11T17:14:44.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:14:45.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:14:45.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:45.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T17:14:45.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:45.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-06-11T17:14:45.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:45.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:14:45.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-06-11T17:14:45.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:45.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-06-11T17:14:45.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-11T17:14:45.905 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672989 2024-06-11T17:14:45.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:46.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-11T17:14:46.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:46.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-06-11T17:14:46.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:46.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:14:46.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-06-11T17:14:46.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:46.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-06-11T17:14:46.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-11T17:14:46.262 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509460 2024-06-11T17:14:46.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:46.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-11T17:14:46.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:14:46.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:46.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:47.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:14:47.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:47.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:47.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:14:47.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:47.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:47.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:47.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:14:47.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:14:47.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:14:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:14:47.762 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:14:47.762 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:48.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:14:48.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:14:48.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:14:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:14:48.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:14:48.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:14:48.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:48.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:14:49.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:14:49.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:14:49.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:14:49.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:14:49.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:14:49.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:14:49.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:14:49.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:14:49.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:14:49.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:14:49.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:14:49.359 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:14:49.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:14:49.360 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:14:49.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:14:49.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:14:49.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:14:49.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:14:49.396 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:49.395+0000 7f2c5bf847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:49.399 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:49.399+0000 7f2c5bf847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:49.401 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:49.399+0000 7f2c5bf847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:14:49.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:14:49.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:14:49.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:14:49.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:14:49.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:14:49.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:49.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:14:49.696 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:14:49.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:49.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:49.938 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:49.935+0000 7f2c5bf847c0 -1 Falling back to public interface 2024-06-11T17:14:50.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:51.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:51.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:52.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:52.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:52.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:14:52.395 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:14:52.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:52.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:52.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:14:53.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:14:53.047+0000 7f2c5bf847c0 -1 osd.0 31 log_to_monitors true 2024-06-11T17:14:53.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:14:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:14:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:14:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:14:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:14:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:14:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1341282060,v1:127.0.0.1:6803/1341282060] [v2:127.0.0.1:6804/1341282060,v1:127.0.0.1:6805/1341282060] exists,up edc62673-a932-407d-a4f6-ec46e7b063f8 2024-06-11T17:14:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:14:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:14:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:14:54.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:14:54.085 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:14:54.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:14:54.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:14:54.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:14:54.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:14:54.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:14:54.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:14:54.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:14:54.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:14:54.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:14:54.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:14:54.523 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:14:54.523 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:14:54.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:14:54.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:54.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:14:54.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855362 2024-06-11T17:14:54.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855362 2024-06-11T17:14:54.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362' 2024-06-11T17:14:54.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:54.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:14:54.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672992 2024-06-11T17:14:54.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672992 2024-06-11T17:14:54.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362 1-42949672992' 2024-06-11T17:14:54.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:14:54.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:14:55.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509463 2024-06-11T17:14:55.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509463 2024-06-11T17:14:55.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362 1-42949672992 2-64424509463' 2024-06-11T17:14:55.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:55.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855362 2024-06-11T17:14:55.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:55.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:14:55.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855362 2024-06-11T17:14:55.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:55.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855362 2024-06-11T17:14:55.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-06-11T17:14:55.239 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 150323855362 2024-06-11T17:14:55.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:14:55.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-11T17:14:55.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:55.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672992 2024-06-11T17:14:55.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:55.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:14:55.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672992 2024-06-11T17:14:55.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:55.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672992 2024-06-11T17:14:55.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-06-11T17:14:55.650 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672992 2024-06-11T17:14:55.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:14:55.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672992 -lt 42949672992 2024-06-11T17:14:55.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:14:55.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509463 2024-06-11T17:14:55.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:14:56.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:14:56.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509463 2024-06-11T17:14:56.001 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:14:56.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509463 2024-06-11T17:14:56.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-06-11T17:14:56.002 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509463 2024-06-11T17:14:56.003 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:56.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509462 -lt 64424509463 2024-06-11T17:14:56.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:14:57.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:14:57.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:14:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509463 -lt 64424509463 2024-06-11T17:14:57.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:14:57.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:57.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:58.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:14:58.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:58.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:58.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:58.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:58.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:58.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:58.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:58.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:14:58.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:58.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:58.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:14:58.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:14:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:14:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:14:58.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:14:58.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:14:58.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:14:58.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:14:58.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:14:58.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:14:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:14:59.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:14:59.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:14:59.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:14:59.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:14:59.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:14:59.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:14:59.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:14:59.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:14:59.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:14:59.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:14:59.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:14:59.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:15:00.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:15:00.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:15:00.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:15:00.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:15:00.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:15:00.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:15:00.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:15:00.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:15:00.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:15:00.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:15:00.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:15:00.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:15:00.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:15:00.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:15:00.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:15:00.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:15:00.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:15:00.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:15:00.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:15:00.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:15:00.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:15:00.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:15:00.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:15:00.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:15:00.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:15:00.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:15:01.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:15:01.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:15:01.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T17:15:01.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:15:01.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T17:15:01.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:15:01.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:15:01.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:15:01.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:15:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:15:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:15:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:15:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:15:02.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:15:02.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:15:02.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:15:02.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:15:02.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:15:02.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:15:02.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:15:02.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:15:02.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:15:02.488 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:14:27.299+0000 7f51e099e640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:15:02.488 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:14:27.299+0000 7f51e099e640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:15:02.488 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:14:27.299+0000 7f51e099e640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-11T17:15:02.489 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:14:27.299+0000 7f51e099e640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-11T17:15:02.489 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:14:27.299+0000 7f51e099e640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:15:02.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:15:02.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:15:02.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:15:02.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:15:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:15:02.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:15:02.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:15:02.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:15:02.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:15:02.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:15:02.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:15:02.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:15:02.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:15:02.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:15:02.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:15:02.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:15:02.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:15:02.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:15:02.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:15:02.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:02.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:02.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:15:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:15:02.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:15:02.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:15:02.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:15:02.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:15:02.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:15:02.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:15:02.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:15:02.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:15:02.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:15:02.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:15:02.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:15:02.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:15:02.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:15:02.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:15:02.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:15:02.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:15:02.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:15:02.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:15:02.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:15:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:15:02.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:15:02.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:15:02.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:02.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:02.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:15:02.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:15:02.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:15:02.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:15:03.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:15:03.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:15:03.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:15:03.319 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:15:03.320 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:03.320 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:03.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:03.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:15:03.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:15:03.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:15:03.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:15:03.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:15:03.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:15:03.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:15:03.540 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:15:03.540 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:15:03.541 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:15:03.541 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:15:03.541 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:03.541 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:03.542 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:15:03.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:15:03.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:15:03.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:15:03.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:15:03.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:15:03.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:15:03.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:15:03.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:15:03.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-06-11T17:15:03.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:15:03.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:15:03.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:15:03.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:15:03.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:15:03.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:15:04.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:15:04.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:15:04.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:15:04.657 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:15:04.658 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:04.658 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:04.658 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:04.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:15:04.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:15:04.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:15:04.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:15:04.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:04.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:15:04.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:15:04.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:15:04.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:15:04.704 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:15:04.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 be9a1541-a93b-434a-86ee-15513821c765' 2024-06-11T17:15:04.705 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:15:04.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYhmhmEQkgKxAAZYEbyGNDxJcK5YmHQ8rK4g== 2024-06-11T17:15:04.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYhmhmEQkgKxAAZYEbyGNDxJcK5YmHQ8rK4g=="}' 2024-06-11T17:15:04.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be9a1541-a93b-434a-86ee-15513821c765 -i td/osd-scrub-repair/0/new.json 2024-06-11T17:15:05.074 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:15:05.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:15:05.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAYhmhmEQkgKxAAZYEbyGNDxJcK5YmHQ8rK4g== --osd-uuid be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:15:05.131 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.127+0000 7f3a1eab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:05.141 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.139+0000 7f3a1eab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.139+0000 7f3a1eab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.143+0000 7f3a1eab37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:15:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.143+0000 7f3a1eab37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:15:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.143+0000 7f3a1eab37c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:15:05.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:05.231+0000 7f3a1eab37c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:15:11.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:15:11.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:15:11.824 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:15:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:15:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:15:12.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:15:12.242 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:15:12.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:15:12.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:15:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:15:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:15:12.272 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:12.271+0000 7fd23ba047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:12.306 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:12.303+0000 7fd23ba047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:12.308 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:12.307+0000 7fd23ba047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:12.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:12.587 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:15:12.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:15:12.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:12.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:12.854 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:12.851+0000 7fd23ba047c0 -1 Falling back to public interface 2024-06-11T17:15:12.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:13.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:13.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:13.929 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:15:13.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:15:13.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:13.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:14.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:15.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:15.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:15.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:15:15.264 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:15:15.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:15.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:15.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:15.471+0000 7fd23ba047c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:15:15.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:16.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:16.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:16.606 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:15:16.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:15:16.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:16.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:16.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:17.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:17.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:17.946 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:15:17.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:15:17.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:17.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:18.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:19.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:19.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:19.325 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:15:19.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:15:19.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:19.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:19.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:20.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:20.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:20.668 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:15:20.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:15:20.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:20.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:21.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:22.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:22.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:22.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:15:22.008 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:15:22.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:22.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:23.342 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:15:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:15:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:23.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:24.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:24.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:24.671 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:15:24.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:15:24.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:24.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:25.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:26.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:26.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:26.007 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:15:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:15:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:26.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:27.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:27.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:27.346 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:15:27.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:15:27.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:27.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:27.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:28.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:28.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:28.683 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:15:28.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:15:28.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:28.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:30.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:30.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:30.300 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:15:30.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:15:30.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:30.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:31.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:32.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:32.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:32.094 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:15:32.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:15:32.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:32.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:32.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:33.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:33.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:33.861 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:15:33.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:15:33.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:33.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:34.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:35.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:35.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:35.689 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:15:35.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:15:35.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:35.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:36.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:37.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:37.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:37.070 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:15:37.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:15:37.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:37.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:37.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:38.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:38.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:38.857 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:15:38.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:15:38.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:38.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:39.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:40.309 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:15:40.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:15:40.644 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3743379836,v1:127.0.0.1:6803/3743379836] [v2:127.0.0.1:6804/3743379836,v1:127.0.0.1:6805/3743379836] exists,up be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:15:40.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:15:40.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:15:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:15:40.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:15:40.651 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:15:40.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c' 2024-06-11T17:15:40.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:15:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8hmhmD1lnKBAAGOeJ3iHpxXo8YkWZuqV/HA== 2024-06-11T17:15:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8hmhmD1lnKBAAGOeJ3iHpxXo8YkWZuqV/HA=="}' 2024-06-11T17:15:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c -i td/osd-scrub-repair/1/new.json 2024-06-11T17:15:41.061 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:15:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:15:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA8hmhmD1lnKBAAGOeJ3iHpxXo8YkWZuqV/HA== --osd-uuid 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:15:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.119+0000 7f6929a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:41.171 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.171+0000 7f6929a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:41.173 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.171+0000 7f6929a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:41.174 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.171+0000 7f6929a137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:15:41.174 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.171+0000 7f6929a137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:15:41.174 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.171+0000 7f6929a137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:15:41.279 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:41.275+0000 7f6929a137c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:15:47.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:15:47.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:15:47.708 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:15:47.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:15:47.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:15:48.131 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:15:48.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:15:48.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:15:48.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:15:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:15:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:15:48.169 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:48.167+0000 7f4142bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:48.200 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:48.199+0000 7f4142bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:48.202 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:48.199+0000 7f4142bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:15:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:15:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:15:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:15:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:15:48.492 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:15:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:15:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:15:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:48.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:48.782 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:48.779+0000 7f4142bd77c0 -1 Falling back to public interface 2024-06-11T17:15:48.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:49.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:49.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:49.886 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:15:49.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:15:49.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:49.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:50.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:51.221 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:15:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:15:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:51.461 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:15:51.459+0000 7f4142bd77c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:15:51.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:52.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:52.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:52.577 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:15:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:15:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:52.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:53.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:53.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:53.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:15:53.912 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:15:53.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:53.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:54.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:15:55.318 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:15:55.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:55.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:55.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:56.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:56.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:56.655 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:15:56.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:15:56.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:56.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:57.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:58.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:58.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:58.054 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:15:58.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:15:58.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:58.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:58.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:15:59.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:15:59.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:15:59.390 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:15:59.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:15:59.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:15:59.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:15:59.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:00.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:00.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:00.721 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:16:00.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:16:00.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:00.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:01.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:02.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:02.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:02.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:16:02.057 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:16:02.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:02.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:02.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:03.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:03.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:03.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:16:03.393 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:16:03.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:03.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:03.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:04.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:04.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:04.732 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:16:04.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:16:04.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:04.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:05.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:06.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:06.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:06.465 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:16:06.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:16:06.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:06.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:07.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:08.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:08.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:08.152 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:16:08.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:16:08.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:08.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:08.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:16:09.544 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:16:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:10.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:11.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:11.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:11.038 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:16:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:16:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:11.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:12.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:12.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:12.627 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:16:12.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:16:12.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:12.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:13.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:14.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:14.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:14.470 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:16:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:16:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:15.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:16.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:16.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:16.174 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:16:16.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:16:16.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:16.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:16:16.552 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1402570349,v1:127.0.0.1:6811/1402570349] [v2:127.0.0.1:6812/1402570349,v1:127.0.0.1:6813/1402570349] exists,up 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:16:16.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:16:16.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:16:16.556 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:16:16.556 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:16:16.556 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:16:16.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:16:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:16:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:16:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:16:16.558 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:16:16.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29c0074f-caff-4d62-8ef5-b85aff11f050 2024-06-11T17:16:16.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 29c0074f-caff-4d62-8ef5-b85aff11f050' 2024-06-11T17:16:16.559 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 29c0074f-caff-4d62-8ef5-b85aff11f050 2024-06-11T17:16:16.559 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:16:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBghmhmNG98IhAAyxAMUwBD3mRFtEChFMPXXg== 2024-06-11T17:16:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBghmhmNG98IhAAyxAMUwBD3mRFtEChFMPXXg=="}' 2024-06-11T17:16:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29c0074f-caff-4d62-8ef5-b85aff11f050 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:16:17.074 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:16:17.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:16:17.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBghmhmNG98IhAAyxAMUwBD3mRFtEChFMPXXg== --osd-uuid 29c0074f-caff-4d62-8ef5-b85aff11f050 2024-06-11T17:16:17.135 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.131+0000 7f9ebec667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:17.167 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.163+0000 7f9ebec667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:17.169 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.167+0000 7f9ebec667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:17.170 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.167+0000 7f9ebec667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:16:17.170 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.167+0000 7f9ebec667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:16:17.170 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.167+0000 7f9ebec667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:16:17.238 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:17.239+0000 7f9ebec667c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:16:23.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:16:23.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:16:23.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:16:23.755 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:16:23.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:16:24.169 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:16:24.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:16:24.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:16:24.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:16:24.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:16:24.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:16:24.206 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:24.202+0000 7ff1fe2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:24.231 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:24.226+0000 7ff1fe2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:24.232 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:24.230+0000 7ff1fe2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:16:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:16:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:16:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:16:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:16:24.518 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:16:24.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:16:24.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:24.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:16:24.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:24.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:24.782 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:24.778+0000 7ff1fe2937c0 -1 Falling back to public interface 2024-06-11T17:16:24.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:25.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:25.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:25.877 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:16:25.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:16:25.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:25.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:26.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:27.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:27.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:27.209 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:16:27.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:16:27.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:27.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:27.451 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:16:27.446+0000 7ff1fe2937c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:16:27.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:28.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:28.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:28.583 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:16:28.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:16:28.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:28.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:28.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:29.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:29.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:29.923 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:16:29.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:16:29.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:29.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:30.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:31.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:31.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:31.332 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:16:31.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:16:31.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:31.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:31.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:32.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:32.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:32.669 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:16:32.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:16:32.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:32.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:34.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:34.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:34.083 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:16:34.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:16:34.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:34.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:34.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:35.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:35.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:35.524 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:16:35.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:16:35.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:35.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:35.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:36.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:36.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:36.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:16:36.906 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:16:36.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:36.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:37.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:38.422 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:16:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:16:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:39.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:39.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:39.762 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:16:39.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:16:39.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:39.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:40.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:41.137 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:16:41.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:16:41.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:41.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:41.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:42.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:42.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:42.793 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:16:42.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:16:42.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:42.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:43.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:44.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:44.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:44.472 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:16:44.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:16:44.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:44.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:45.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:46.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:46.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:46.294 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:16:46.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:16:46.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:46.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:47.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:48.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:48.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:48.165 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:16:48.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:16:48.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:48.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:49.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:50.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:50.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:50.031 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:16:50.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:16:50.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:50.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:50.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:51.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:51.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:51.484 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:16:51.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:16:51.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:51.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:16:53.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:16:53.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:16:53.379 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:16:53.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:16:53.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:16:53.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:16:53.756 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3150701241,v1:127.0.0.1:6819/3150701241] [v2:127.0.0.1:6820/3150701241,v1:127.0.0.1:6821/3150701241] exists,up 29c0074f-caff-4d62-8ef5-b85aff11f050 2024-06-11T17:16:53.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:16:53.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:16:53.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:16:53.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-06-11T17:16:53.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-11T17:16:54.220 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T17:16:54.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:16:55.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-06-11T17:16:56.089 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T17:16:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-11T17:16:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:16:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:16:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:16:56.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:16:56.114 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:16:56.114 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:16:56.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:16:56.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:16:56.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:16:56.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:16:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:16:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:16:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:16:56.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:16:56.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:16:56.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:16:56.758 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:16:56.758 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:16:56.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:16:56.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:16:56.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:16:57.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-06-11T17:16:57.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-06-11T17:16:57.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-06-11T17:16:57.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:16:57.016 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:16:57.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-06-11T17:16:57.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-06-11T17:16:57.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2024-06-11T17:16:57.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:16:57.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:16:57.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509443 2024-06-11T17:16:57.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509443 2024-06-11T17:16:57.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509443' 2024-06-11T17:16:57.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:16:57.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-06-11T17:16:57.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:16:57.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:16:57.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-06-11T17:16:57.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:16:57.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-06-11T17:16:57.309 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836499 2024-06-11T17:16:57.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-11T17:16:57.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:16:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836499 2024-06-11T17:16:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:16:58.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:16:58.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:16:58.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-11T17:16:58.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:16:58.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-06-11T17:16:58.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:16:58.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:16:58.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-06-11T17:16:58.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:16:58.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-06-11T17:16:58.965 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672978 2024-06-11T17:16:58.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-11T17:16:58.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:16:59.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-11T17:16:59.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:16:59.314 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509443 2024-06-11T17:16:59.315 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:16:59.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:16:59.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509443 2024-06-11T17:16:59.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:16:59.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509443 2024-06-11T17:16:59.319 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509443 2024-06-11T17:16:59.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-06-11T17:16:59.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:16:59.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509443 -lt 64424509443 2024-06-11T17:16:59.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:16:59.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:16:59.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:00.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:17:00.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:17:00.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:17:00.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:17:00.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:17:00.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:00.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:00.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:17:00.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:17:00.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:17:00.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-06-11T17:17:00.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-06-11T17:17:00.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-06-11T17:17:00.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:00.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:01.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:01.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:02.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:02.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:02.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:02.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:02.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-11T17:17:02.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:17:02.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-11T17:17:02.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:17:02.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:17:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:17:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:17:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:17:02.497 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:17:02.497 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:17:02.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:17:02.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:17:02.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:17:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:17:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:17:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:17:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:03.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-11T17:17:05.133 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-11T17:17:05.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:17:05.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:17:05.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:05.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:05.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:17:05.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:17:05.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:17:05.675 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:17:05.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:05.676 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:17:05.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:17:05.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:17:05.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:17:05.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:17:05.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:05.714+0000 7f9625c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:05.718+0000 7f9625c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:05.718+0000 7f9625c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:06.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:17:06.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:17:06.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:06.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:06.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:06.254+0000 7f9625c627c0 -1 Falling back to public interface 2024-06-11T17:17:06.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:07.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:17:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:17:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:08.983 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:08.982+0000 7f9625c627c0 -1 osd.0 21 log_to_monitors true 2024-06-11T17:17:09.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:10.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:10.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:10.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:17:10.045 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:17:10.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:10.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:10.414 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1921773725,v1:127.0.0.1:6803/1921773725] [v2:127.0.0.1:6804/1921773725,v1:127.0.0.1:6805/1921773725] exists,up be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:17:10.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:17:10.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:17:10.415 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:17:10.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:17:10.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:17:10.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:17:10.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:17:10.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:17:10.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:17:10.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:17:10.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:17:10.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:17:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:17:10.954 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:10.954 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:17:10.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:17:10.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:10.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:17:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182402 2024-06-11T17:17:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182402 2024-06-11T17:17:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402' 2024-06-11T17:17:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:11.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:17:11.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-06-11T17:17:11.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-06-11T17:17:11.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672981' 2024-06-11T17:17:11.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:11.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:17:11.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-06-11T17:17:11.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-06-11T17:17:11.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672981 2-64424509447' 2024-06-11T17:17:11.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:11.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182402 2024-06-11T17:17:11.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:11.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:17:11.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182402 2024-06-11T17:17:11.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:11.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182402 2024-06-11T17:17:11.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-11T17:17:11.410 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 107374182402 2024-06-11T17:17:11.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:11.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 107374182402 2024-06-11T17:17:11.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:17:12.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:17:12.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:13.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-11T17:17:13.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:13.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-06-11T17:17:13.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:13.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:17:13.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:13.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-06-11T17:17:13.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-06-11T17:17:13.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-11T17:17:13.108 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672981 2024-06-11T17:17:13.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:13.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-11T17:17:13.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:13.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-06-11T17:17:13.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:13.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:17:13.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-06-11T17:17:13.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:13.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-06-11T17:17:13.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-11T17:17:13.461 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509447 2024-06-11T17:17:13.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:13.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-11T17:17:13.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:17:13.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:13.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:14.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:17:14.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:17:14.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:17:14.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:17:14.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:17:14.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:17:14.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:17:14.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:17:14.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:17:14.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:17:14.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:14.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:15.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:17:15.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:17:15.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:17:15.008 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:17:15.009 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:17:15.009 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:17:15.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:17:15.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:17:15.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:17:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:17:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:17:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:16.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-11T17:17:17.673 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:104778fc:::obj2:head# 2024-06-11T17:17:18.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:18.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:17:18.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:17:18.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:17:18.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:17:18.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:17:18.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:17:18.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:17:18.215 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:17:18.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:18.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:17:18.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:17:18.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:17:18.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:17:18.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:17:18.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:18.254+0000 7ff3279327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:18.256 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:18.258+0000 7ff3279327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:18.258 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:18.258+0000 7ff3279327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:18.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:17:18.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:18.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:18.794 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:18.794+0000 7ff3279327c0 -1 Falling back to public interface 2024-06-11T17:17:18.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:19.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:19.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:19.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:17:19.891 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:19.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:19.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:20.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:21.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:21.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:21.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:17:21.225 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:17:21.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:21.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:21.438 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:21.438+0000 7ff3279327c0 -1 osd.0 26 log_to_monitors true 2024-06-11T17:17:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:22.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:22.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:22.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:17:22.567 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:17:22.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:22.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/528851607,v1:127.0.0.1:6803/528851607] [v2:127.0.0.1:6804/528851607,v1:127.0.0.1:6805/528851607] exists,up be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:17:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:17:23.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:17:23.017 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:17:23.017 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:17:23.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:17:23.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:17:23.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:17:23.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:17:23.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:17:23.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:17:23.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:17:23.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:17:23.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:17:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:17:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:23.547 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:17:23.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:17:23.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:23.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:17:23.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T17:17:23.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T17:17:23.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882' 2024-06-11T17:17:23.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:23.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:17:23.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-11T17:17:23.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-11T17:17:23.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-06-11T17:17:23.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:23.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:17:23.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-06-11T17:17:23.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-06-11T17:17:23.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509450' 2024-06-11T17:17:23.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:23.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018882 2024-06-11T17:17:23.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:23.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:17:24.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018882 2024-06-11T17:17:24.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:24.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T17:17:24.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-11T17:17:24.001 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 128849018882 2024-06-11T17:17:24.002 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:24.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T17:17:24.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:24.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-11T17:17:24.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:24.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:17:24.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-11T17:17:24.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:24.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-11T17:17:24.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-11T17:17:24.322 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672985 2024-06-11T17:17:24.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:24.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672985 2024-06-11T17:17:24.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:17:25.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:17:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:26.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-06-11T17:17:26.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:26.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-06-11T17:17:26.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:26.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:17:26.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:26.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-06-11T17:17:26.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-06-11T17:17:26.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-11T17:17:26.034 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509450 2024-06-11T17:17:26.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:26.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509450 2024-06-11T17:17:26.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:17:26.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:26.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:26.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:17:26.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:17:26.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:17:26.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:17:26.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:17:26.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:17:26.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:17:26.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:17:27.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:17:27.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:17:27.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:27.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:27.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:17:27.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:17:27.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:17:27.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-11T17:17:27.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:17:27.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:17:27.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:17:27.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:17:27.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:17:27.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:17:27.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:17:27.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:17:27.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:17:28.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:17:28.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:17:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-11T17:17:30.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:30.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:17:30.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:17:30.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:30.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:30.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:17:30.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:17:30.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:17:30.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:17:30.271 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:17:30.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:30.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:17:30.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:17:30.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:17:30.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:17:30.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:17:30.308 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:30.310+0000 7f5bed4a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:30.310 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:30.310+0000 7f5bed4a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:30.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:30.314+0000 7f5bed4a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:30.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:17:30.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:17:30.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:30.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:17:30.845 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:30.846+0000 7f5bed4a17c0 -1 Falling back to public interface 2024-06-11T17:17:30.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:17:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:31.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:31.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:17:32.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:33.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:33.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:33.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:17:33.286 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:17:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:17:33.567 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:33.566+0000 7f5bed4a17c0 -1 osd.1 31 log_to_monitors true 2024-06-11T17:17:33.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:34.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:34.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:34.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:17:34.629 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:17:34.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:34.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1052800357,v1:127.0.0.1:6811/1052800357] [v2:127.0.0.1:6812/1052800357,v1:127.0.0.1:6813/1052800357] exists,up 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:17:34.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:17:34.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:17:34.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:17:34.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:17:34.989 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:17:34.989 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:17:34.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:17:34.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:17:34.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:17:35.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:17:35.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:17:35.425 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:35.425 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:17:35.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:17:35.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:35.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:17:35.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018885 2024-06-11T17:17:35.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018885 2024-06-11T17:17:35.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885' 2024-06-11T17:17:35.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:35.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:17:35.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855362 2024-06-11T17:17:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855362 2024-06-11T17:17:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-06-11T17:17:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:35.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:17:35.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509454' 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018885 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:35.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:17:35.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018885 2024-06-11T17:17:35.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:35.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018885 2024-06-11T17:17:35.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-06-11T17:17:35.893 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 128849018885 2024-06-11T17:17:35.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:36.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018885 -lt 128849018885 2024-06-11T17:17:36.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:36.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855362 2024-06-11T17:17:36.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:36.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:17:36.218 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855362 2024-06-11T17:17:36.218 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:36.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855362 2024-06-11T17:17:36.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-11T17:17:36.219 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 150323855362 2024-06-11T17:17:36.219 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:36.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-11T17:17:36.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:36.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:36.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-06-11T17:17:36.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:17:36.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-06-11T17:17:36.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:36.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-06-11T17:17:36.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-11T17:17:36.572 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509454 2024-06-11T17:17:36.572 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:36.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509454 2024-06-11T17:17:36.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:17:37.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:17:37.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:38.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-11T17:17:38.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:17:38.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:38.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:38.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:17:38.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:17:38.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:17:38.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:17:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:17:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:39.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:17:39.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-11T17:17:39.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:17:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-06-11T17:17:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-06-11T17:17:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:17:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T17:17:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T17:17:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:17:40.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-06-11T17:17:40.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-06-11T17:17:40.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:40.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:40.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:40.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-06-11T17:17:40.633 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:17:40.633 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:17:40.633 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T17:17:40.633 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-05-28T17:16:00.636549+0000" 2024-06-11T17:17:40.633 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:17:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T17:17:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:17:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:40.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:40.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:16:54.128447+0000 '>' 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:40.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:17:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:17:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:17:41.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:17:41.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:41.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:42.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:16:54.128447+0000 '>' 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:42.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:17:43.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:17:43.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:17:43.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:17:43.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:43.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:43.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:43.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:43.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:16:54.128447+0000 '>' 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:43.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:17:44.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:17:44.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:17:44.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:17:44.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:44.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:44.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:44.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:44.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:16:54.128447+0000 '>' 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:44.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:17:45.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:17:45.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:17:45.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:17:45.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:17:45.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:17:45.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:17:45.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:17:41.866107+0000 '>' 2024-06-11T17:16:54.128447+0000 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:17:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:17:46.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:17:46.322 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:17:46.322 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:17:46.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:17:46.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:17:46.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:17:46.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:17:46.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:17:46.760 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:46.760 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:17:46.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:17:46.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:46.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:17:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018889 2024-06-11T17:17:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018889 2024-06-11T17:17:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018889' 2024-06-11T17:17:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:46.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:17:47.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855365 2024-06-11T17:17:47.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855365 2024-06-11T17:17:47.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018889 1-150323855365' 2024-06-11T17:17:47.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:47.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:17:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018889 1-150323855365 2-64424509457' 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018889 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:47.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:17:47.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018889 2024-06-11T17:17:47.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:47.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018889 2024-06-11T17:17:47.199 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 128849018889 2024-06-11T17:17:47.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018889' 2024-06-11T17:17:47.200 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:47.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018889 2024-06-11T17:17:47.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:17:48.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:17:48.506 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:48.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018889 -lt 128849018889 2024-06-11T17:17:48.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:48.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:48.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855365 2024-06-11T17:17:48.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:17:48.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855365 2024-06-11T17:17:48.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:48.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855365 2024-06-11T17:17:48.834 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 150323855365 2024-06-11T17:17:48.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-06-11T17:17:48.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:49.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-11T17:17:49.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:49.199 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:49.199 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-06-11T17:17:49.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:17:49.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:49.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-06-11T17:17:49.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-06-11T17:17:49.203 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509457 2024-06-11T17:17:49.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-11T17:17:49.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:49.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-11T17:17:49.559 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:17:49.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:49.561 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:49.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:17:49.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:17:49.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:17:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:17:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:17:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:17:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:17:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:17:50.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:17:50.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:17:50.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:17:50.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:17:50.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:17:51.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:17:51.054 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:51.054 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:17:51.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:17:51.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:51.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:17:51.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018890 2024-06-11T17:17:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018890 2024-06-11T17:17:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018890' 2024-06-11T17:17:51.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:51.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:17:51.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855367 2024-06-11T17:17:51.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855367 2024-06-11T17:17:51.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-06-11T17:17:51.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:17:51.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:17:51.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509459 2024-06-11T17:17:51.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509459 2024-06-11T17:17:51.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509459' 2024-06-11T17:17:51.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:51.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018890 2024-06-11T17:17:51.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:51.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:17:51.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018890 2024-06-11T17:17:51.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:51.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018890 2024-06-11T17:17:51.517 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 128849018890 2024-06-11T17:17:51.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-06-11T17:17:51.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:51.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018889 -lt 128849018890 2024-06-11T17:17:51.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:17:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:17:52.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:17:53.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018891 -lt 128849018890 2024-06-11T17:17:53.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:53.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855367 2024-06-11T17:17:53.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:53.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:17:53.173 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855367 2024-06-11T17:17:53.173 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:53.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855367 2024-06-11T17:17:53.175 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 150323855367 2024-06-11T17:17:53.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-06-11T17:17:53.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:17:53.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855367 -lt 150323855367 2024-06-11T17:17:53.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:17:53.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:17:53.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509459 2024-06-11T17:17:53.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:17:53.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-06-11T17:17:53.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:17:53.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509459 2024-06-11T17:17:53.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-11T17:17:53.518 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509459 2024-06-11T17:17:53.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:17:53.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-11T17:17:53.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:17:53.857 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-11T17:17:53.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-06-11T17:17:53.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-11T17:17:54.177 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:17:54.178 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-11T17:17:41.866498+0000 21'10 36:101 [1,0] 1 [1,0] 1 21'10 2024-06-11T17:17:41.866107+0000 21'10 2024-06-11T17:17:41.866107+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:17:41.866107+0000 10 0 2024-06-11T17:17:54.178 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:17:54.178 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:17:54.178 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:17:54.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-11T17:17:54.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-06-11T17:17:54.497 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:17:54.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-06-11T17:17:54.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:17:54.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-11T17:17:54.517 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-11T17:17:54.517 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:17:54.517 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T17:17:54.518 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:17:54.518 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T17:17:54.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:17:54.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-11T17:17:54.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:17:55.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-06-11T17:17:55.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:17:55.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:17:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:17:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:17:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:56.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-06-11T17:17:56.586 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:17:56.587 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:17:56.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:17:56.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:17:56.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:17:56.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:17:56.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:17:56.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:17:56.889 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:17:56.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:17:56.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:17:56.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:17:56.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:17:56.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:17:56.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:17:56.926 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:56.926+0000 7f6ef53c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:56.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:56.930+0000 7f6ef53c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:56.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:56.934+0000 7f6ef53c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:57.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:17:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:17:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:17:57.466+0000 7f6ef53c47c0 -1 Falling back to public interface 2024-06-11T17:17:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:58.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:58.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:58.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:17:58.580 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:17:58.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:58.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:17:58.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:17:59.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:17:59.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:17:59.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:17:59.912 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:17:59.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:17:59.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:00.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:00.683 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:00.686+0000 7f6ef53c47c0 -1 osd.0 36 log_to_monitors true 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:01.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:01.694 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/3750789530,v1:127.0.0.1:6803/3750789530] [v2:127.0.0.1:6804/3750789530,v1:127.0.0.1:6805/3750789530] exists,up be9a1541-a93b-434a-86ee-15513821c765 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:18:01.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:18:01.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:18:01.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:18:02.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:18:02.304 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:18:02.304 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:18:02.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:18:02.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:02.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:18:02.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691842 2024-06-11T17:18:02.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691842 2024-06-11T17:18:02.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691842' 2024-06-11T17:18:02.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:02.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:18:02.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855370 2024-06-11T17:18:02.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855370 2024-06-11T17:18:02.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-06-11T17:18:02.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:02.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:18:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509462 2024-06-11T17:18:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509462 2024-06-11T17:18:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509462' 2024-06-11T17:18:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:02.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-171798691842 2024-06-11T17:18:02.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:02.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:18:02.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-171798691842 2024-06-11T17:18:02.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:02.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691842 2024-06-11T17:18:02.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-06-11T17:18:02.762 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 171798691842 2024-06-11T17:18:02.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:03.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 171798691842 2024-06-11T17:18:03.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:18:04.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:18:04.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:04.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-11T17:18:04.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:04.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855370 2024-06-11T17:18:04.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:04.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:18:04.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:04.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855370 2024-06-11T17:18:04.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855370 2024-06-11T17:18:04.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-06-11T17:18:04.432 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 150323855370 2024-06-11T17:18:04.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:18:04.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855370 -lt 150323855370 2024-06-11T17:18:04.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:04.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509462 2024-06-11T17:18:04.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:04.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:18:04.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509462 2024-06-11T17:18:04.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:04.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509462 2024-06-11T17:18:04.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-11T17:18:04.783 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509462 2024-06-11T17:18:04.783 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:18:05.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509462 -lt 64424509462 2024-06-11T17:18:05.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:18:05.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:18:05.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:18:05.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:18:05.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:18:05.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:18:05.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:18:05.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:18:05.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:18:05.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:18:05.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:18:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:18:05.879 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:18:05.879 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:18:05.879 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:18:06.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:18:06.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:18:06.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:18:06.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-06-11T17:18:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:18:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:18:06.295 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-11T17:18:06.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-11T17:18:06.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:18:06.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:18:06.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:18:06.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:18:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-06-11T17:18:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:18:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:18:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:18:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:18:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:18:06.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:18:06.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:18:06.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-06-11T17:18:08.402 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-06-11T17:18:08.402 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:104778fc:::obj2:head# 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:18:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:18:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:18:08.953 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:18:08.953 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:18:08.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:18:08.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:18:08.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:18:08.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:18:08.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:18:08.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:18:08.993 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:08.994+0000 7f41c9e7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:08.996 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:08.998+0000 7f41c9e7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:08.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:08.998+0000 7f41c9e7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:09.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:18:09.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:09.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:18:09.538 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:09.538+0000 7f41c9e7f7c0 -1 Falling back to public interface 2024-06-11T17:18:09.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:18:10.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:11.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:11.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:11.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:18:11.987 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:18:11.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:11.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:18:12.259 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:12.258+0000 7f41c9e7f7c0 -1 osd.1 41 log_to_monitors true 2024-06-11T17:18:12.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:13.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:13.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:13.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:18:13.369 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:18:13.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:13.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/2631112194,v1:127.0.0.1:6811/2631112194] [v2:127.0.0.1:6812/2631112194,v1:127.0.0.1:6813/2631112194] exists,up 32941f31-5d3b-4c3e-ac1d-a92d8c9e865c 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:18:13.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:18:13.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:18:13.705 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:18:13.705 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:18:13.705 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:18:13.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:18:13.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:18:13.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:18:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:18:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:18:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:18:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:18:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:18:13.814 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:18:14.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:18:14.146 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:18:14.146 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:18:14.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:18:14.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:14.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:18:14.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691845 2024-06-11T17:18:14.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691845 2024-06-11T17:18:14.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691845' 2024-06-11T17:18:14.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:14.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:18:14.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528322 2024-06-11T17:18:14.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528322 2024-06-11T17:18:14.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-06-11T17:18:14.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:14.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:18:14.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509466 2024-06-11T17:18:14.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509466 2024-06-11T17:18:14.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509466' 2024-06-11T17:18:14.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:14.583 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-171798691845 2024-06-11T17:18:14.583 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:14.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:18:14.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-171798691845 2024-06-11T17:18:14.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:14.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691845 2024-06-11T17:18:14.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-06-11T17:18:14.586 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 171798691845 2024-06-11T17:18:14.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:14.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691844 -lt 171798691845 2024-06-11T17:18:14.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:18:15.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:18:15.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:16.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691845 -lt 171798691845 2024-06-11T17:18:16.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:16.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-193273528322 2024-06-11T17:18:16.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:16.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:18:16.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:16.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-193273528322 2024-06-11T17:18:16.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528322 2024-06-11T17:18:16.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-06-11T17:18:16.263 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 193273528322 2024-06-11T17:18:16.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:18:16.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528322 -lt 193273528322 2024-06-11T17:18:16.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:16.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:16.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509466 2024-06-11T17:18:16.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:18:16.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-06-11T17:18:16.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:16.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509466 2024-06-11T17:18:16.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-06-11T17:18:16.637 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509466 2024-06-11T17:18:16.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:18:17.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509466 -lt 64424509466 2024-06-11T17:18:17.006 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:18:17.007 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:18:17.007 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:18:17.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:18:17.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:18:17.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:18:17.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:18:17.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:18:17.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:18:17.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:18:17.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:18:17.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:18:17.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:18:17.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:18:17.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:18:18.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:18:18.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:18:18.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:18:18.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-06-11T17:18:18.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-06-11T17:18:18.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-06-11T17:18:18.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:18.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:18.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:18.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:18.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:18.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-06-11T17:18:18.813 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-11T17:18:18.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:18.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T17:18:18.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:18.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:18:18.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:18:18.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:18:18.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:18:18.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:18.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:18.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:18.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:19.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:17:41.866107+0000 '>' 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:19.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:18:20.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:18:20.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:18:20.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:18:20.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:20.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:20.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:20.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:20.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:17:41.866107+0000 '>' 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:20.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:18:21.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:18:21.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:18:21.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:18:21.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:21.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:21.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:21.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:21.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:17:41.866107+0000 '>' 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:21.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:18:22.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:18:22.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:18:22.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:18:22.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:22.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:22.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:22.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:23.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:17:41.866107+0000 '>' 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:23.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:18:24.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:18:24.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:18:24.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:18:24.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:18:24.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:18:24.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:18:24.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:18:24.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:18:19.376340+0000 '>' 2024-06-11T17:17:41.866107+0000 2024-06-11T17:18:24.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:18:24.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-06-11T17:18:26.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-11T17:18:26.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:18:26.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:18:26.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:18:26.847 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:18:26.847 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:18:26.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:18:26.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:26.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:18:27.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691849 2024-06-11T17:18:27.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691849 2024-06-11T17:18:27.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691849' 2024-06-11T17:18:27.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:27.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:18:27.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528325 2024-06-11T17:18:27.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528325 2024-06-11T17:18:27.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2024-06-11T17:18:27.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:18:27.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:18:27.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-06-11T17:18:27.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-06-11T17:18:27.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509469' 2024-06-11T17:18:27.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:27.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-171798691849 2024-06-11T17:18:27.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:27.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:18:27.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-171798691849 2024-06-11T17:18:27.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:27.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691849 2024-06-11T17:18:27.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-06-11T17:18:27.306 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 171798691849 2024-06-11T17:18:27.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:27.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691847 -lt 171798691849 2024-06-11T17:18:27.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:18:28.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:18:28.629 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:18:28.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691849 -lt 171798691849 2024-06-11T17:18:28.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:28.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-193273528325 2024-06-11T17:18:28.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:28.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:18:28.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-193273528325 2024-06-11T17:18:28.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:28.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528325 2024-06-11T17:18:28.975 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 193273528325 2024-06-11T17:18:28.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-06-11T17:18:28.975 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:18:29.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528326 -lt 193273528325 2024-06-11T17:18:29.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:18:29.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:18:29.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-06-11T17:18:29.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:18:29.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:18:29.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-06-11T17:18:29.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-06-11T17:18:29.312 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509469 2024-06-11T17:18:29.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-06-11T17:18:29.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:18:29.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-06-11T17:18:29.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-11T17:18:29.953 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:18:29.953 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-06-11T17:18:19.376698+0000 21'10 46:138 [1,0] 1 [1,0] 1 21'10 2024-06-11T17:18:19.376340+0000 21'10 2024-06-11T17:18:19.376340+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:18:19.376340+0000 9 0 2024-06-11T17:18:29.954 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:18:29.954 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:18:29.954 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:18:29.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-11T17:18:29.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-06-11T17:18:30.274 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:18:30.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:18:30.293 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:18:30.293 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:18:30.293 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.514+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:17:41.862+0000 7f5bd142b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:18:19.374+0000 7f41ade09640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:18:19.374+0000 7f41ade09640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:18:19.374+0000 7f41ade09640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:18:19.374+0000 7f41ade09640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-06-11T17:18:30.294 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:18:19.374+0000 7f41ade09640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:18:30.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:18:30.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:18:30.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:18:30.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:18:30.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:18:30.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:18:30.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:18:30.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:18:30.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:18:30.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:18:30.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:18:30.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:18:30.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:18:30.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:18:30.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:18:30.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:18:30.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:18:30.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:30.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:30.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:18:30.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:18:30.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:18:30.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:18:30.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:18:30.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:18:30.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:18:30.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:18:30.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:18:30.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:18:30.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:18:30.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:18:30.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:18:30.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:18:30.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:18:30.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:18:30.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:18:30.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:18:30.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:18:30.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:30.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:30.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:18:30.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:18:30.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:18:30.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:18:30.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:18:30.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:30.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-06-11T17:18:30.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:18:30.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:18:30.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:18:30.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:18:30.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:18:30.914 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:18:30.914 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:30.914 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:30.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:30.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:18:31.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:18:31.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:18:31.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:18:31.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:18:31.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:18:31.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:18:31.135 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:18:31.135 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:18:31.135 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:18:31.136 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:18:31.136 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:31.136 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:31.136 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:18:31.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:18:31.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:18:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:18:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:18:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:18:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:18:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:18:31.258 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:18:31.258 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:18:31.258 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:18:31.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:18:31.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-06-11T17:18:31.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:18:31.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:18:31.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:18:31.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:18:31.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:18:31.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:18:32.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:18:32.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:18:32.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:18:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:18:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:32.262 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:32.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:32.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:18:32.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:18:32.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:18:32.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-06-11T17:18:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-11T17:18:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:18:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:18:32.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:18:32.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:18:32.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:18:32.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:18:32.306 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:18:32.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:18:32.308 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:18:32.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8683bf91-3a3c-4745-9621-8bda794b88ed' 2024-06-11T17:18:32.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:18:32.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDohmhmQRu1ExAAt0GrO3Y0MhYHPA2MuQnThg== 2024-06-11T17:18:32.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDohmhmQRu1ExAAt0GrO3Y0MhYHPA2MuQnThg=="}' 2024-06-11T17:18:32.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8683bf91-3a3c-4745-9621-8bda794b88ed -i td/osd-scrub-repair/0/new.json 2024-06-11T17:18:32.594 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:18:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:18:32.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDohmhmQRu1ExAAt0GrO3Y0MhYHPA2MuQnThg== --osd-uuid 8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:18:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.654+0000 7fe307cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:32.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.654+0000 7fe307cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:32.655 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.658+0000 7fe307cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:32.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.658+0000 7fe307cbe7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:18:32.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.658+0000 7fe307cbe7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:18:32.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.658+0000 7fe307cbe7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:18:32.728 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:32.730+0000 7fe307cbe7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:18:39.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:18:39.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:18:39.190 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:18:39.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:18:39.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:18:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:18:39.610 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:18:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:18:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:18:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:18:39.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:18:39.645 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:39.646+0000 7fd4371ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:39.683 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:39.686+0000 7fd4371ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:39.685 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:39.686+0000 7fd4371ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:18:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:39.972 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:18:39.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:18:39.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:39.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:40.245 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:40.246+0000 7fd4371ef7c0 -1 Falling back to public interface 2024-06-11T17:18:40.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:41.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:41.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:41.341 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:18:41.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:18:41.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:41.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:41.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:42.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:42.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:42.668 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:18:42.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:18:42.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:42.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:42.816 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:18:42.818+0000 7fd4371ef7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:18:43.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:44.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:44.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:44.010 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:18:44.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:18:44.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:44.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:44.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:45.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:45.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:45.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:18:45.415 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:18:45.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:45.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:45.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:46.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:46.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:46.826 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:18:46.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:18:46.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:46.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:47.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:48.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:48.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:48.167 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:18:48.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:18:48.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:48.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:48.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:49.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:49.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:49.627 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:18:49.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:18:49.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:49.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:49.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:50.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:50.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:50.961 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:18:50.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:18:50.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:50.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:51.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:52.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:52.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:52.289 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:18:52.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:18:52.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:52.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:52.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:53.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:53.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:53.783 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:18:53.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:18:53.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:53.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:54.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:55.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:55.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:55.116 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:18:55.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:18:55.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:55.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:55.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:56.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:56.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:56.453 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:18:56.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:18:56.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:56.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:56.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:57.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:57.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:57.982 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:18:57.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:18:57.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:57.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:18:58.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:18:59.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:18:59.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:18:59.731 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:18:59.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:18:59.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:18:59.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:00.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:01.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:01.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:01.476 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:19:01.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:19:01.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:01.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:01.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:02.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:02.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:02.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:19:02.884 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:19:02.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:02.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:03.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:04.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:04.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:04.385 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:19:04.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:19:04.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:04.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:04.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:05.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:05.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:05.953 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:19:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:19:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:06.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:07.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:07.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:07.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:19:07.583 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:19:07.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:07.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:19:07.931 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1044100790,v1:127.0.0.1:6803/1044100790] [v2:127.0.0.1:6804/1044100790,v1:127.0.0.1:6805/1044100790] exists,up 8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:19:07.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:19:07.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:19:07.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:19:07.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:19:07.939 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:19:07.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:19:07.939 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:19:07.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0' 2024-06-11T17:19:07.940 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:19:07.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALh2hmpP4UORAAFgTyAOClVdPqyXMOq5B25Q== 2024-06-11T17:19:07.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALh2hmpP4UORAAFgTyAOClVdPqyXMOq5B25Q=="}' 2024-06-11T17:19:07.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 -i td/osd-scrub-repair/1/new.json 2024-06-11T17:19:08.467 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:19:08.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:19:08.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQALh2hmpP4UORAAFgTyAOClVdPqyXMOq5B25Q== --osd-uuid 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:19:08.527 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.529+0000 7f5de8bb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:08.568 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.569+0000 7f5de8bb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:08.570 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.569+0000 7f5de8bb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:08.571 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.569+0000 7f5de8bb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:19:08.571 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.569+0000 7f5de8bb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:19:08.571 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.573+0000 7f5de8bb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:19:08.653 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:08.653+0000 7f5de8bb47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:19:15.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:19:15.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:19:15.147 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:19:15.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:19:15.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:19:15.602 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:19:15.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:19:15.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:19:15.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:19:15.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:19:15.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:19:15.640 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:15.641+0000 7ff3b0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:15.664 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:15.665+0000 7ff3b0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:15.666 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:15.665+0000 7ff3b0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:15.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:19:15.938 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:19:15.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:15.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:16.213 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:16.213+0000 7ff3b0a2b7c0 -1 Falling back to public interface 2024-06-11T17:19:16.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:17.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:17.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:17.282 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:19:17.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:19:17.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:17.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:18.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:18.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:18.610 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:19:18.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:19:18.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:18.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:18.836 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:18.837+0000 7ff3b0a2b7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:19:18.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:19.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:19.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:19.956 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:19:19.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:19:19.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:19.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:20.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:21.297 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:19:21.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:19:21.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:21.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:21.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:22.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:22.979 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:19:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:19:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:24.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:24.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:24.045 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:19:24.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:19:24.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:24.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:25.375 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:19:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:19:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:25.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:26.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:26.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:26.789 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:19:26.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:19:26.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:26.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:27.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:28.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:28.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:28.126 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:19:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:19:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:28.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:29.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:29.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:29.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:19:29.498 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:19:29.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:29.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:29.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:30.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:30.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:30.952 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:19:30.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:19:30.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:30.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:31.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:32.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:32.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:32.290 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:19:32.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:19:32.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:32.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:33.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:34.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:34.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:34.116 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:19:34.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:19:34.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:34.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:35.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:35.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:35.895 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:19:35.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:19:35.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:35.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:36.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:37.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:37.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:37.583 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:19:37.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:19:37.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:37.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:38.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:39.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:39.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:39.203 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:19:39.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:19:39.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:39.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:40.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:41.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:41.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:41.042 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:19:41.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:19:41.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:41.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:41.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:42.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:42.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:42.463 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:19:42.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:19:42.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:42.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:43.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:44.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:44.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:44.013 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:19:44.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:19:44.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:44.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:19:44.351 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3557812170,v1:127.0.0.1:6811/3557812170] [v2:127.0.0.1:6812/3557812170,v1:127.0.0.1:6813/3557812170] exists,up 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:19:44.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:19:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:19:44.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:19:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:19:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:19:44.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:19:44.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6ecf0ae-5463-4c69-aba8-7b0bade8ad07 2024-06-11T17:19:44.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b6ecf0ae-5463-4c69-aba8-7b0bade8ad07' 2024-06-11T17:19:44.359 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 b6ecf0ae-5463-4c69-aba8-7b0bade8ad07 2024-06-11T17:19:44.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:19:44.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwh2hmIZLQFhAAFRHkqwK0NJ+xo3HKRPKIRw== 2024-06-11T17:19:44.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwh2hmIZLQFhAAFRHkqwK0NJ+xo3HKRPKIRw=="}' 2024-06-11T17:19:44.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6ecf0ae-5463-4c69-aba8-7b0bade8ad07 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:19:44.756 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:19:44.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:19:44.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAwh2hmIZLQFhAAFRHkqwK0NJ+xo3HKRPKIRw== --osd-uuid b6ecf0ae-5463-4c69-aba8-7b0bade8ad07 2024-06-11T17:19:44.814 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.813+0000 7f32a554d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:44.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.857+0000 7f32a554d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:44.860 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.861+0000 7f32a554d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:44.861 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.861+0000 7f32a554d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:19:44.861 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.861+0000 7f32a554d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:19:44.861 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.861+0000 7f32a554d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:19:44.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:44.933+0000 7f32a554d7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:19:51.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:19:51.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:19:51.966 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:19:51.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:19:51.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:19:52.387 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:19:52.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:19:52.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:19:52.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:19:52.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:19:52.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:19:52.424 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:52.425+0000 7f63483317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:52.446 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:52.445+0000 7f63483317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:52.447 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:52.449+0000 7f63483317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:19:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:19:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:19:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:19:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:19:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:19:52.730 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:19:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:19:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:52.997 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:52.997+0000 7f63483317c0 -1 Falling back to public interface 2024-06-11T17:19:53.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:54.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:54.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:54.062 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:19:54.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:19:54.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:54.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:54.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:55.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:55.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:55.389 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:19:55.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:19:55.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:55.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:55.787 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:19:55.789+0000 7f63483317c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:19:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:56.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:56.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:56.852 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:19:56.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:19:56.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:56.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:57.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:58.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:58.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:58.193 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:19:58.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:19:58.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:58.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:58.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:19:59.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:19:59.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:19:59.552 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:19:59.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:19:59.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:19:59.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:19:59.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:00.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:00.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:00.973 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:20:00.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:20:00.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:00.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:01.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:02.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:02.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:02.306 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:20:02.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:20:02.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:02.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:02.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:03.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:03.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:03.811 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:20:03.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:20:03.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:03.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:04.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:05.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:05.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:05.149 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:20:05.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:20:05.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:05.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:05.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:06.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:06.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:06.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:20:06.511 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:20:06.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:06.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:06.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:07.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:07.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:07.991 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:20:07.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:20:07.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:07.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:08.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:09.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:09.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:09.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:20:09.325 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:20:09.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:09.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:10.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:11.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:11.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:11.154 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:20:11.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:20:11.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:11.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:12.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:12.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:12.832 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:20:12.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:20:12.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:12.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:13.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:14.423 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:20:14.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:20:14.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:14.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:14.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:15.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:15.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:15.846 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:20:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:20:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:16.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:17.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:17.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:17.685 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:20:17.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:20:17.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:17.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:18.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:19.531 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:20:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:20:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:20.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:21.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:21.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:21.440 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:20:21.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:20:21.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:21.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:20:21.775 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/715850356,v1:127.0.0.1:6819/715850356] [v2:127.0.0.1:6820/715850356,v1:127.0.0.1:6821/715850356] exists,up b6ecf0ae-5463-4c69-aba8-7b0bade8ad07 2024-06-11T17:20:21.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:20:21.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:20:21.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:20:21.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-06-11T17:20:21.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-11T17:20:22.262 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T17:20:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:20:23.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-06-11T17:20:24.044 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T17:20:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-11T17:20:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:20:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:20:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:20:24.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:20:24.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:24.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:20:24.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836501 2024-06-11T17:20:24.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836501 2024-06-11T17:20:24.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501' 2024-06-11T17:20:24.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:24.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:20:24.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672972 2024-06-11T17:20:24.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672972 2024-06-11T17:20:24.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-06-11T17:20:24.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:24.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:20:25.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509447' 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836501 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:20:25.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836501 2024-06-11T17:20:25.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836501 2024-06-11T17:20:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-11T17:20:25.070 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836501 2024-06-11T17:20:25.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:20:25.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-11T17:20:25.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:25.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672972 2024-06-11T17:20:25.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:25.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:20:25.529 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672972 2024-06-11T17:20:25.529 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:25.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672972 2024-06-11T17:20:25.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-11T17:20:25.530 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672972 2024-06-11T17:20:25.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:20:25.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-11T17:20:25.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:25.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:25.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-06-11T17:20:25.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:20:25.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-06-11T17:20:25.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:25.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-06-11T17:20:25.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-11T17:20:25.876 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509447 2024-06-11T17:20:25.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:20:26.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-11T17:20:26.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:20:26.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:26.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:20:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:26.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:26.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:20:26.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:26.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:26.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:27.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:20:27.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-11T17:20:27.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:20:27.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-11T17:20:27.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:20:27.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:20:27.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:20:27.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:27.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:27.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:27.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:27.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:27.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:27.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:27.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:20:27.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:27.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:27.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:28.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:20:28.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:20:28.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:20:28.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:20:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:20:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:20:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:20:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:20:28.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:20:28.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:28.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:29.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:20:29.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:29.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:29.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:29.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:20:29.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:20:29.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:20:29.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:20:29.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:20:29.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:20:29.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:20:29.538 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:20:29.538 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:20:29.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:20:29.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:20:29.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T17:20:29.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:20:29.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T17:20:30.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:20:30.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:30.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:30.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:30.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:30.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:30.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:30.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:30.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:20:30.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:30.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:30.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:20:31.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:20:31.039 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:20:31.039 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:20:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:20:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:20:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-06-11T17:20:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:20:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-06-11T17:20:32.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:20:32.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:32.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:32.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:32.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:32.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:32.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:32.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:32.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:20:32.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:32.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:32.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-06-11T17:20:32.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-06-11T17:20:32.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:32.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:33.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:33.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:33.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:33.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:33.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:33.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:33.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:33.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:33.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:33.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:34.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:34.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:34.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:34.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:34.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:34.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:34.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-11T17:20:34.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-11T17:20:34.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-11T17:20:34.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:20:34.631 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:20:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:20:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:20:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:20:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:20:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:20:34.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:20:34.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:20:34.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:20:35.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:20:35.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:20:35.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:20:35.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:20:35.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:20:35.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-11T17:20:37.100 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:20:37.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:20:37.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:20:37.642 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:20:37.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:20:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:20:37.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:20:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:20:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:20:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:20:37.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:20:37.680 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:37.681+0000 7fa48d78a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:37.684 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:37.685+0000 7fa48d78a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:37.685 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:37.685+0000 7fa48d78a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:37.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:20:37.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:37.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:20:38.228 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:38.229+0000 7fa48d78a7c0 -1 Falling back to public interface 2024-06-11T17:20:38.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:39.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:39.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:39.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:20:39.353 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:20:39.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:39.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:20:39.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:20:40.856 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:40.857+0000 7fa48d78a7c0 -1 osd.0 21 log_to_monitors true 2024-06-11T17:20:41.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:20:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:20:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:42.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:20:42.450 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3292509130,v1:127.0.0.1:6803/3292509130] [v2:127.0.0.1:6804/3292509130,v1:127.0.0.1:6805/3292509130] exists,up 8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:20:42.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:20:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:20:42.452 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:20:42.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:20:42.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:20:42.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:20:42.571 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:42.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:20:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182402 2024-06-11T17:20:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182402 2024-06-11T17:20:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402' 2024-06-11T17:20:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:43.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:20:43.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-06-11T17:20:43.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-06-11T17:20:43.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-06-11T17:20:43.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:43.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509452' 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182402 2024-06-11T17:20:43.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:43.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:20:43.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182402 2024-06-11T17:20:43.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182402 2024-06-11T17:20:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-11T17:20:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 107374182402 2024-06-11T17:20:43.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:20:43.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-11T17:20:43.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:43.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-06-11T17:20:43.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:43.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:20:43.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-06-11T17:20:43.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:43.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-06-11T17:20:43.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-11T17:20:43.683 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672976 2024-06-11T17:20:43.683 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:20:44.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-11T17:20:44.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:44.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-06-11T17:20:44.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:44.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:20:44.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-06-11T17:20:44.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-06-11T17:20:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-06-11T17:20:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509452 2024-06-11T17:20:44.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:20:44.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509452 2024-06-11T17:20:44.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:20:45.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:20:45.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:20:45.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509452 2024-06-11T17:20:45.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:20:45.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:45.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:46.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:20:46.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:46.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:46.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:46.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:46.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:46.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:46.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:46.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:20:46.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:46.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:46.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:46.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:20:46.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:20:46.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:20:46.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-11T17:20:46.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:20:46.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:20:46.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:20:46.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:20:47.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:20:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:20:48.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:20:48.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-11T17:20:48.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:20:48.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:20:48.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:20:48.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:20:48.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:20:48.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-06-11T17:20:50.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:20:50.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:20:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:20:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:20:50.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:20:50.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:20:50.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:20:50.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:20:50.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:20:50.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:20:50.495 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:50.497+0000 7fe24ce387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:50.497+0000 7fe24ce387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:50.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:50.501+0000 7fe24ce387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:20:50.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:20:50.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:51.041 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:51.041+0000 7fe24ce387c0 -1 Falling back to public interface 2024-06-11T17:20:51.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:52.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:52.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:52.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:20:52.146 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:20:52.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:52.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:20:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:53.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:53.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:53.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:20:53.482 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:20:53.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:53.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:20:53.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:20:54.317 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:20:54.317+0000 7fe24ce387c0 -1 osd.1 26 log_to_monitors true 2024-06-11T17:20:54.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:20:54.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:20:54.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:20:54.866 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:20:54.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:20:54.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1454995925,v1:127.0.0.1:6811/1454995925] [v2:127.0.0.1:6812/1454995925,v1:127.0.0.1:6813/1454995925] exists,up 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:20:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:20:55.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:20:55.347 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:20:55.347 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:20:55.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:20:55.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:20:55.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:20:55.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:20:55.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:20:55.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:20:55.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:20:55.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:20:55.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:20:55.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:20:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:20:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:20:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:20:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:55.814 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:20:55.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182405 2024-06-11T17:20:55.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182405 2024-06-11T17:20:55.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405' 2024-06-11T17:20:55.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:55.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:20:56.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T17:20:56.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T17:20:56.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-06-11T17:20:56.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:20:56.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:20:56.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-06-11T17:20:56.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-06-11T17:20:56.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509455' 2024-06-11T17:20:56.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:56.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182405 2024-06-11T17:20:56.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:56.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:20:56.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182405 2024-06-11T17:20:56.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:56.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182405 2024-06-11T17:20:56.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-06-11T17:20:56.300 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 107374182405 2024-06-11T17:20:56.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:20:56.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182404 -lt 107374182405 2024-06-11T17:20:56.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:20:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:20:57.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:20:57.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182406 -lt 107374182405 2024-06-11T17:20:57.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:57.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-128849018882 2024-06-11T17:20:57.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:57.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:20:57.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-128849018882 2024-06-11T17:20:57.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:57.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T17:20:57.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-06-11T17:20:57.966 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 128849018882 2024-06-11T17:20:57.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:20:58.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T17:20:58.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:20:58.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-06-11T17:20:58.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:20:58.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:20:58.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-06-11T17:20:58.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:20:58.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-06-11T17:20:58.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-11T17:20:58.304 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509455 2024-06-11T17:20:58.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:20:58.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509455 2024-06-11T17:20:58.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:20:58.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:58.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:59.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:20:59.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:20:59.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:20:59.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:20:59.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:20:59.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:20:59.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:20:59.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:20:59.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:20:59.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:20:59.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:20:59.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:20:59.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:20:59.851 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:20:59.851 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:20:59.851 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:20:59.851 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:20:59.852 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:21:00.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:21:00.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:21:00.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:21:00.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:21:00.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:21:00.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:21:00.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:21:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:21:00.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:21:00.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-11T17:21:00.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:21:00.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:21:00.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:21:00.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:21:00.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:21:00.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-11T17:21:02.290 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:104778fc:::obj2:head# 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:21:02.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:21:02.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:21:02.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:21:02.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:21:02.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:21:02.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:21:02.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:21:02.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:21:02.870 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:02.869+0000 7f040abbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:02.873 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:02.873+0000 7f040abbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:02.875 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:02.877+0000 7f040abbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:21:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:21:03.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:03.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:03.424 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:03.425+0000 7f040abbe7c0 -1 Falling back to public interface 2024-06-11T17:21:03.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:04.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:21:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:04.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:05.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:05.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:05.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:21:05.910 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:21:05.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:05.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:05.988 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:05.989+0000 7f040abbe7c0 -1 osd.0 31 log_to_monitors true 2024-06-11T17:21:06.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:07.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:07.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:07.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:21:07.251 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:21:07.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:07.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:07.611 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/2386241331,v1:127.0.0.1:6803/2386241331] [v2:127.0.0.1:6804/2386241331,v1:127.0.0.1:6805/2386241331] exists,up 8683bf91-3a3c-4745-9621-8bda794b88ed 2024-06-11T17:21:07.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:21:07.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:21:07.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:21:07.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:21:07.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:21:07.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:21:07.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:21:07.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:21:07.613 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:21:07.613 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:21:07.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:21:07.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:21:07.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:21:07.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:21:08.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:21:08.049 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:08.050 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:21:08.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:21:08.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:08.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:21:08.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855362 2024-06-11T17:21:08.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855362 2024-06-11T17:21:08.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362' 2024-06-11T17:21:08.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:08.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:21:08.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018885 2024-06-11T17:21:08.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018885 2024-06-11T17:21:08.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-06-11T17:21:08.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:08.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:21:08.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509459 2024-06-11T17:21:08.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509459 2024-06-11T17:21:08.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509459' 2024-06-11T17:21:08.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:08.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855362 2024-06-11T17:21:08.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:08.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:21:08.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855362 2024-06-11T17:21:08.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:08.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855362 2024-06-11T17:21:08.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-06-11T17:21:08.514 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 150323855362 2024-06-11T17:21:08.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:08.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 150323855362 2024-06-11T17:21:08.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:21:09.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:21:09.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:10.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-11T17:21:10.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:10.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-128849018885 2024-06-11T17:21:10.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:10.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:21:10.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-128849018885 2024-06-11T17:21:10.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:10.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018885 2024-06-11T17:21:10.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-06-11T17:21:10.181 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 128849018885 2024-06-11T17:21:10.182 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:21:10.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018885 -lt 128849018885 2024-06-11T17:21:10.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:10.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509459 2024-06-11T17:21:10.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:10.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:21:10.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-06-11T17:21:10.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509459 2024-06-11T17:21:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-11T17:21:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509459 2024-06-11T17:21:10.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:21:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-11T17:21:10.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:21:10.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:10.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:11.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:21:11.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:21:11.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:21:11.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:21:11.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:21:11.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:11.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:12.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:21:12.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:21:12.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:21:12.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-11T17:21:12.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:21:12.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:21:12.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:21:12.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:21:12.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:21:12.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:21:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:21:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:21:12.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-11T17:21:14.167 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:21:14.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:21:14.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:21:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:21:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:21:14.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:21:14.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:21:14.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:21:14.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:21:14.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:21:14.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:21:14.754 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:14.753+0000 7fc2d67fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:14.756 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:14.757+0000 7fc2d67fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:14.758 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:14.757+0000 7fc2d67fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:21:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:21:15.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:21:15.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:15.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:15.317+0000 7fc2d67fb7c0 -1 Falling back to public interface 2024-06-11T17:21:15.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:16.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:16.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:16.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:21:16.428 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:16.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:16.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:21:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:17.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:21:17.980 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:17.981+0000 7fc2d67fb7c0 -1 osd.1 36 log_to_monitors true 2024-06-11T17:21:18.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:21:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:21:19.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:19.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/46196986,v1:127.0.0.1:6811/46196986] [v2:127.0.0.1:6812/46196986,v1:127.0.0.1:6813/46196986] exists,up 2d6b0b51-533f-4640-b6a1-6f9e7097d6b0 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:21:19.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:21:19.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:21:19.468 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:21:19.468 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:21:19.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:21:19.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:21:19.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:21:19.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:21:19.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:21:19.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:21:19.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:21:19.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:21:19.585 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:21:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:21:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:19.917 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:21:19.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:21:19.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:19.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:21:20.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855365 2024-06-11T17:21:20.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855365 2024-06-11T17:21:20.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855365' 2024-06-11T17:21:20.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:20.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:21:20.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691842 2024-06-11T17:21:20.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691842 2024-06-11T17:21:20.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-06-11T17:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509462 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509462 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509462' 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855365 2024-06-11T17:21:20.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:20.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:21:20.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855365 2024-06-11T17:21:20.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:20.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855365 2024-06-11T17:21:20.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-06-11T17:21:20.397 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 150323855365 2024-06-11T17:21:20.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:20.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855364 -lt 150323855365 2024-06-11T17:21:20.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:21:21.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:21:21.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:22.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855366 -lt 150323855365 2024-06-11T17:21:22.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:22.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-171798691842 2024-06-11T17:21:22.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:22.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:21:22.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-171798691842 2024-06-11T17:21:22.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:22.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691842 2024-06-11T17:21:22.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-06-11T17:21:22.059 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 171798691842 2024-06-11T17:21:22.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:21:22.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-11T17:21:22.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:22.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509462 2024-06-11T17:21:22.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:22.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:21:22.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509462 2024-06-11T17:21:22.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:22.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509462 2024-06-11T17:21:22.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-11T17:21:22.397 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509462 2024-06-11T17:21:22.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:21:22.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509462 -lt 64424509462 2024-06-11T17:21:22.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:21:22.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:22.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:23.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:21:23.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:21:23.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:21:23.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:21:23.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:21:23.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:23.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:21:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:21:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:21:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-11T17:21:24.047 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: { 2024-06-11T17:21:24.047 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-11T17:21:24.047 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:21:24.060 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: { 2024-06-11T17:21:24.060 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-11T17:21:24.060 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:21:24.073 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2: { 2024-06-11T17:21:24.074 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-11T17:21:24.074 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:21:24.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-06-11T17:21:24.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T17:21:24.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-11T17:21:24.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-11T17:21:24.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:21:24.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-06-11T17:21:24.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-06-11T17:21:24.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:21:24.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T17:21:24.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:21:24.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T17:21:24.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-06-11T17:21:24.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-06-11T17:21:24.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:21:24.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:21:24.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:21:24.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:21:25.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:25.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-06-11T17:21:25.262 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:21:25.263 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": true, 2024-06-11T17:21:25.263 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T17:21:25.263 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-05-28T17:19:45.266261+0000" 2024-06-11T17:21:25.263 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:21:25.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:21:25.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:21:25.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:21:25.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:21:25.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:21:25.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:20:22.196861+0000 '>' 2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:25.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:21:26.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:21:26.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:21:26.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:21:26.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:21:26.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:21:26.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:21:26.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:21:26.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:20:22.196861+0000 '>' 2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:26.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:21:27.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:21:27.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:21:27.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:21:27.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:21:27.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:21:27.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:21:27.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:21:28.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:20:22.196861+0000 '>' 2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:28.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:21:29.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:21:29.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:21:29.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:21:29.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:21:29.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:21:29.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:21:29.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:21:29.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:21:28.024028+0000 '>' 2024-06-11T17:20:22.196861+0000 2024-06-11T17:21:29.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:21:29.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-11T17:21:29.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:21:29.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:21:29.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:21:29.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:21:29.654 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:21:29.654 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:21:29.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:21:29.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:21:29.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:21:29.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:21:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:21:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:21:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:21:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:30.101 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:21:30.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855368 2024-06-11T17:21:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855368 2024-06-11T17:21:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855368' 2024-06-11T17:21:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:30.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:21:30.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691845 2024-06-11T17:21:30.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691845 2024-06-11T17:21:30.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-06-11T17:21:30.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:30.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:21:30.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509465 2024-06-11T17:21:30.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509465 2024-06-11T17:21:30.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509465' 2024-06-11T17:21:30.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:30.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855368 2024-06-11T17:21:30.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:30.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:21:30.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855368 2024-06-11T17:21:30.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:30.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855368 2024-06-11T17:21:30.600 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 150323855368 2024-06-11T17:21:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-06-11T17:21:30.600 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:30.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855367 -lt 150323855368 2024-06-11T17:21:30.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:21:31.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:21:31.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:32.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855369 -lt 150323855368 2024-06-11T17:21:32.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:32.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-171798691845 2024-06-11T17:21:32.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:32.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:21:32.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-171798691845 2024-06-11T17:21:32.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:32.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691845 2024-06-11T17:21:32.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-06-11T17:21:32.246 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 171798691845 2024-06-11T17:21:32.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:21:32.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691845 -lt 171798691845 2024-06-11T17:21:32.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:32.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:32.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509465 2024-06-11T17:21:32.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:21:32.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509465 2024-06-11T17:21:32.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:32.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509465 2024-06-11T17:21:32.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-06-11T17:21:32.620 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509465 2024-06-11T17:21:32.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:21:32.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509465 -lt 64424509465 2024-06-11T17:21:32.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:21:32.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:32.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:33.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:21:33.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:21:33.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:21:33.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:21:33.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:21:33.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:21:33.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:21:33.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:21:33.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:21:33.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:21:33.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:21:33.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:21:34.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:21:34.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:21:34.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:21:34.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-06-11T17:21:34.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:21:34.101 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:21:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:21:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:21:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:21:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:21:34.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:34.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:21:34.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855370 2024-06-11T17:21:34.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855370 2024-06-11T17:21:34.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855370' 2024-06-11T17:21:34.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:34.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:21:34.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691847 2024-06-11T17:21:34.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691847 2024-06-11T17:21:34.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-06-11T17:21:34.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:21:34.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509467 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509467 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509467' 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855370 2024-06-11T17:21:34.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:34.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:21:34.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855370 2024-06-11T17:21:34.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855370 2024-06-11T17:21:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-06-11T17:21:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 150323855370 2024-06-11T17:21:34.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:35.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855369 -lt 150323855370 2024-06-11T17:21:35.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:21:36.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:21:36.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:21:36.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855370 -lt 150323855370 2024-06-11T17:21:36.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:36.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:36.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-171798691847 2024-06-11T17:21:36.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:21:36.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-171798691847 2024-06-11T17:21:36.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:36.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691847 2024-06-11T17:21:36.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-06-11T17:21:36.547 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 171798691847 2024-06-11T17:21:36.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:21:36.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691847 -lt 171798691847 2024-06-11T17:21:36.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:21:36.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509467 2024-06-11T17:21:36.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:21:36.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:21:36.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509467 2024-06-11T17:21:36.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:21:36.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509467 2024-06-11T17:21:36.894 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509467 2024-06-11T17:21:36.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-06-11T17:21:36.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:21:37.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509467 -lt 64424509467 2024-06-11T17:21:37.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-06-11T17:21:37.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-11T17:21:37.593 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:21:37.593 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-11T17:21:28.024427+0000 21'10 41:97 [1,0] 1 [1,0] 1 21'10 2024-06-11T17:21:28.024028+0000 21'10 2024-06-11T17:21:28.024028+0000 0 1 periodic scrub scheduled @ 2024-06-12T17:21:28.024028+0000 10 0 2024-06-11T17:21:37.593 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:21:37.593 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:21:37.594 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:21:37.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-11T17:21:37.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-06-11T17:21:37.909 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:21:37.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:21:37.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:21:37.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:21:37.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:21:37.926 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:21:37.926 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:21:37.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:21:37.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:21:37.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:21:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:21:38.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:21:38.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:21:38.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:21:38.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:21:38.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:21:38.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:21:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:21:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:21:38.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:21:38.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:21:38.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:21:38.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:21:38.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:21:38.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:21:38.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:38.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:38.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:21:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:21:38.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:21:38.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:21:38.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:21:38.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:21:38.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:21:38.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:21:38.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:21:38.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:21:38.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:21:38.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:21:38.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:21:38.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:21:38.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:21:38.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:21:38.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:21:38.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:21:38.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:21:38.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:21:38.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:21:38.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:21:38.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:21:38.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:21:38.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:21:38.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:38.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:38.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:21:38.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:21:38.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:21:38.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:21:38.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:21:38.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:38.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:38.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:21:38.292 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:21:38.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:21:38.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:21:38.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:21:38.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:21:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:21:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:38.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:21:38.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:21:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:21:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:21:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:21:38.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:21:38.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:21:38.991 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:21:38.991 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:21:38.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:21:39.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:21:39.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:21:39.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:21:39.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:21:39.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:21:39.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:21:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:21:40.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:21:40.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:21:40.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:21:40.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:21:40.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:40.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:40.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:40.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:21:40.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:21:40.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:21:40.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:21:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:21:40.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:21:40.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:21:40.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:21:40.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5e07f7e6-5f12-4c91-afaf-f8d057014c5c' 2024-06-11T17:21:40.155 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:21:40.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:21:40.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkh2hm9yh5ChAAR4KF4z90PhaNcXRVcGtAmA== 2024-06-11T17:21:40.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkh2hm9yh5ChAAR4KF4z90PhaNcXRVcGtAmA=="}' 2024-06-11T17:21:40.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e07f7e6-5f12-4c91-afaf-f8d057014c5c -i td/osd-scrub-repair/0/new.json 2024-06-11T17:21:40.433 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:21:40.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:21:40.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCkh2hm9yh5ChAAR4KF4z90PhaNcXRVcGtAmA== --osd-uuid 5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:21:40.488 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.488+0000 7fcab922a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:40.500 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.500+0000 7fcab922a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:40.502 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.500+0000 7fcab922a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:40.502 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.500+0000 7fcab922a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:21:40.503 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.500+0000 7fcab922a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:21:40.503 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.500+0000 7fcab922a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:21:40.601 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:40.600+0000 7fcab922a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:21:47.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:21:47.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:21:47.129 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:21:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:21:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:21:47.544 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:21:47.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:21:47.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:21:47.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:21:47.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:21:47.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:21:47.581 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:47.580+0000 7ff3b4d617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:47.616 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:47.616+0000 7ff3b4d617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:47.617 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:47.616+0000 7ff3b4d617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:21:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:21:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:21:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:21:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:21:47.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:21:47.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:47.899 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:21:47.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:21:47.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:47.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:48.181 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:48.180+0000 7ff3b4d617c0 -1 Falling back to public interface 2024-06-11T17:21:48.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:49.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:49.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:49.266 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:21:49.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:21:49.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:49.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:49.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:50.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:50.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:50.602 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:21:50.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:21:50.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:50.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:50.917 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:21:50.916+0000 7ff3b4d617c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:21:50.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:51.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:51.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:51.942 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:21:51.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:21:51.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:51.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:52.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:53.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:53.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:53.280 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:21:53.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:21:53.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:53.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:53.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:54.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:54.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:54.726 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:21:54.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:21:54.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:54.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:55.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:56.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:56.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:56.060 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:21:56.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:21:56.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:56.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:56.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:57.392 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:21:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:21:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:57.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:21:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:21:58.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:21:58.733 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:21:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:21:58.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:21:58.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:21:59.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:00.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:22:00.067 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:22:00.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:00.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:00.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:01.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:01.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:01.396 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:22:01.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:22:01.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:01.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:01.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:02.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:02.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:02.783 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:22:02.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:22:02.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:02.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:03.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:04.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:04.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:04.122 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:22:04.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:22:04.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:04.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:04.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:05.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:05.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:05.680 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:22:05.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:22:05.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:05.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:06.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:07.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:07.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:07.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:22:07.522 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:22:07.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:07.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:08.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:09.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:09.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:09.352 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:22:09.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:22:09.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:09.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:10.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:11.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:11.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:11.190 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:22:11.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:22:11.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:11.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:11.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:12.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:12.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:12.690 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:22:12.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:22:12.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:12.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:13.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:14.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:14.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:14.261 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:22:14.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:22:14.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:14.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:14.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:15.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:15.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:15.790 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:22:15.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:22:15.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:15.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:22:16.160 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3944091233,v1:127.0.0.1:6803/3944091233] [v2:127.0.0.1:6804/3944091233,v1:127.0.0.1:6805/3944091233] exists,up 5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:22:16.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:22:16.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:22:16.166 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:22:16.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:22:16.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:22:16.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:22:16.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:22:16.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd1fdd86-bd7a-4695-87e0-0ec67d1addc2 2024-06-11T17:22:16.168 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 fd1fdd86-bd7a-4695-87e0-0ec67d1addc2 2024-06-11T17:22:16.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fd1fdd86-bd7a-4695-87e0-0ec67d1addc2' 2024-06-11T17:22:16.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:22:16.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIh2hmzMJ7CxAARrh0WMuUkw8kLlsu4TuRuQ== 2024-06-11T17:22:16.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIh2hmzMJ7CxAARrh0WMuUkw8kLlsu4TuRuQ=="}' 2024-06-11T17:22:16.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd1fdd86-bd7a-4695-87e0-0ec67d1addc2 -i td/osd-scrub-repair/1/new.json 2024-06-11T17:22:16.575 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:22:16.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:22:16.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDIh2hmzMJ7CxAARrh0WMuUkw8kLlsu4TuRuQ== --osd-uuid fd1fdd86-bd7a-4695-87e0-0ec67d1addc2 2024-06-11T17:22:16.638 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.636+0000 7ff0267537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:16.676 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.676+0000 7ff0267537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:16.677 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.676+0000 7ff0267537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:16.678 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.676+0000 7ff0267537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:22:16.678 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.676+0000 7ff0267537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:22:16.678 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.676+0000 7ff0267537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:22:16.760 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:16.760+0000 7ff0267537c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:22:23.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:22:23.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:22:23.418 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:22:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:22:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:22:23.833 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:22:23.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:22:23.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:22:23.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:22:23.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:22:23.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:22:23.872 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:23.872+0000 7fb38ac7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:23.898 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:23.896+0000 7fb38ac7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:23.900+0000 7fb38ac7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:22:24.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:24.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:22:24.195 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:22:24.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:24.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:24.452 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:24.452+0000 7fb38ac7b7c0 -1 Falling back to public interface 2024-06-11T17:22:24.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:25.547 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:22:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:22:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:25.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:26.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:26.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:26.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:22:26.883 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:22:26.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:26.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:27.117 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:27.116+0000 7fb38ac7b7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:22:27.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:28.225 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:22:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:22:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:28.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:29.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:29.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:29.566 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:22:29.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:22:29.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:29.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:29.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:30.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:30.903 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:22:30.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:22:30.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:30.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:31.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:32.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:32.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:32.374 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:22:32.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:22:32.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:32.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:32.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:33.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:33.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:33.579 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:22:33.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:22:33.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:34.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:35.126 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:22:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:22:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:36.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:36.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:36.568 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:22:36.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:22:36.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:36.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:37.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:38.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:38.027 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:22:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:22:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:38.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:38.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:39.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:39.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:39.368 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:22:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:22:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:39.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:40.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:40.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:40.786 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:22:40.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:22:40.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:40.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:41.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:42.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:42.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:42.241 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:22:42.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:22:42.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:42.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:42.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:43.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:43.838 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:22:43.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:22:43.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:43.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:44.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:45.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:45.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:45.611 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:22:45.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:22:45.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:45.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:46.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:47.073 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:22:47.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:22:47.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:47.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:47.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:48.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:48.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:48.529 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:22:48.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:22:48.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:48.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:49.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:50.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:50.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:50.141 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:22:50.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:22:50.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:50.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:50.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:51.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:51.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:51.730 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:22:51.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:22:51.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:51.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:52.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:22:53.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:22:53.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:22:53.543 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:22:53.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:22:53.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:22:53.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:22:53.879 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3946425964,v1:127.0.0.1:6811/3946425964] [v2:127.0.0.1:6812/3946425964,v1:127.0.0.1:6813/3946425964] exists,up fd1fdd86-bd7a-4695-87e0-0ec67d1addc2 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:22:53.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:22:53.884 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:22:53.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:22:53.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:22:53.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:22:53.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:22:53.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b16f5060-7cb1-445e-9b47-51983c628d88 2024-06-11T17:22:53.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b16f5060-7cb1-445e-9b47-51983c628d88' 2024-06-11T17:22:53.887 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 b16f5060-7cb1-445e-9b47-51983c628d88 2024-06-11T17:22:53.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:22:53.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDth2hmv5NGNhAAhOc8H5TcV5N3SCCi7OQyYA== 2024-06-11T17:22:53.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDth2hmv5NGNhAAhOc8H5TcV5N3SCCi7OQyYA=="}' 2024-06-11T17:22:53.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b16f5060-7cb1-445e-9b47-51983c628d88 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:22:54.332 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:22:54.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:22:54.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDth2hmv5NGNhAAhOc8H5TcV5N3SCCi7OQyYA== --osd-uuid b16f5060-7cb1-445e-9b47-51983c628d88 2024-06-11T17:22:54.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.392+0000 7f0b3d1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:54.425 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.424+0000 7f0b3d1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:54.426 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.428+0000 7f0b3d1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:22:54.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.428+0000 7f0b3d1df7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:22:54.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.428+0000 7f0b3d1df7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:22:54.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.428+0000 7f0b3d1df7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:22:54.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:22:54.500+0000 7f0b3d1df7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:23:01.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:23:01.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:23:01.197 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:23:01.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:23:01.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:23:01.610 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:23:01.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:23:01.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:23:01.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:23:01.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:23:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:23:01.647 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:01.648+0000 7f6fd877c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:01.679 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:01.680+0000 7f6fd877c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:01.681 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:01.680+0000 7f6fd877c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:01.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:23:01.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:23:01.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:23:01.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:23:01.962 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:23:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:23:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:23:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:02.222 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:02.220+0000 7f6fd877c7c0 -1 Falling back to public interface 2024-06-11T17:23:02.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:03.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:03.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:03.312 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:23:03.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:23:03.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:03.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:03.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:04.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:04.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:04.640 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:23:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:23:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:04.918 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:04.916+0000 7f6fd877c7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:23:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:05.979 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:23:05.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:23:05.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:05.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:06.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:07.318 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:23:07.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:23:07.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:07.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:08.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:08.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:08.662 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:23:08.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:23:08.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:08.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:09.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:10.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:10.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:10.014 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:23:10.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:23:10.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:10.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:10.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:11.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:11.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:11.351 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:23:11.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:23:11.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:11.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:11.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:12.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:12.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:12.927 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:23:12.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:23:12.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:12.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:13.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:14.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:14.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:14.262 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:23:14.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:23:14.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:14.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:14.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:15.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:15.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:15.677 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:23:15.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:23:15.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:15.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:16.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:17.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:17.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:17.156 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:23:17.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:23:17.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:17.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:17.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:18.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:18.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:18.571 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:23:18.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:23:18.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:18.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:18.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:19.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:19.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:19.910 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:23:19.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:23:19.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:19.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:20.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:21.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:21.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:21.398 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:23:21.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:23:21.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:21.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:22.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:23.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:23.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:23.511 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:23:23.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:23:23.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:23.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:25.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:25.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:25.100 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:23:25.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:23:25.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:25.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:26.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:27.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:27.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:27.129 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:23:27.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:23:27.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:27.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:28.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:29.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:29.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:29.010 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:23:29.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:23:29.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:29.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:29.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:30.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:30.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:30.860 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:23:30.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:23:30.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:30.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:23:31.224 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3440478357,v1:127.0.0.1:6819/3440478357] [v2:127.0.0.1:6820/3440478357,v1:127.0.0.1:6821/3440478357] exists,up b16f5060-7cb1-445e-9b47-51983c628d88 2024-06-11T17:23:31.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:23:31.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:23:31.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:23:31.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-06-11T17:23:31.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-11T17:23:31.714 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T17:23:31.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:23:32.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-06-11T17:23:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T17:23:33.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-11T17:23:33.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:23:33.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:23:33.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:23:33.538 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:23:33.539 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:23:33.539 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:23:33.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:23:33.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:23:33.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:23:33.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:23:33.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:23:33.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:23:33.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:23:33.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:23:33.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:23:33.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:23:33.995 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:23:33.996 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:23:33.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:23:33.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:33.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:23:34.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-06-11T17:23:34.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-06-11T17:23:34.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-06-11T17:23:34.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:34.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:23:34.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-06-11T17:23:34.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-06-11T17:23:34.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-06-11T17:23:34.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:34.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:23:34.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-06-11T17:23:34.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-06-11T17:23:34.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509449' 2024-06-11T17:23:34.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:34.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-06-11T17:23:34.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:34.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:23:34.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-06-11T17:23:34.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:34.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-06-11T17:23:34.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-06-11T17:23:34.471 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836500 2024-06-11T17:23:34.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:23:34.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836500 2024-06-11T17:23:34.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:23:35.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:23:35.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:23:36.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836500 2024-06-11T17:23:36.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:36.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-06-11T17:23:36.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:36.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:23:36.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:36.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-06-11T17:23:36.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-06-11T17:23:36.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-11T17:23:36.127 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672977 2024-06-11T17:23:36.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:23:36.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-06-11T17:23:36.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:36.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-06-11T17:23:36.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:36.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:23:36.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:36.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-06-11T17:23:36.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-06-11T17:23:36.499 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509449 2024-06-11T17:23:36.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-11T17:23:36.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:23:36.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-06-11T17:23:36.859 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:23:36.860 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:23:36.860 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:23:37.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:23:37.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:23:37.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:23:37.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:23:37.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:23:37.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:23:37.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:23:38.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:23:38.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:23:38.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:23:38.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-06-11T17:23:38.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-06-11T17:23:38.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:23:38.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-11T17:23:38.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:23:38.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:23:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:23:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:23:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:23:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:23:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:23:38.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:23:38.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:23:38.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:23:38.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:23:38.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:23:38.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:23:38.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:23:38.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:23:38.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:23:38.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:23:38.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:23:38.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:23:38.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:23:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:23:39.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-11T17:23:40.702 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:23:41.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:23:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:23:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:23:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:23:41.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:23:41.242 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:23:41.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:23:41.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:23:41.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:23:41.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:23:41.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:23:41.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:23:41.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:41.280+0000 7f7ede42a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:41.283 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:41.284+0000 7f7ede42a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:41.284 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:41.284+0000 7f7ede42a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:23:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:41.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:23:41.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:41.816+0000 7f7ede42a7c0 -1 Falling back to public interface 2024-06-11T17:23:41.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:42.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:42.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:42.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:23:42.954 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:23:42.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:42.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:23:43.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:23:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:23:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:23:44.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:23:44.499+0000 7f7ede42a7c0 -1 osd.0 21 log_to_monitors true 2024-06-11T17:23:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:23:45.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:23:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:23:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:23:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:23:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:23:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:23:45.979 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/755214843,v1:127.0.0.1:6803/755214843] [v2:127.0.0.1:6804/755214843,v1:127.0.0.1:6805/755214843] exists,up 5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:23:45.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:23:45.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:23:45.981 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:23:45.981 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:23:45.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:23:45.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:23:45.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:23:46.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:46.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:23:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182402 2024-06-11T17:23:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182402 2024-06-11T17:23:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402' 2024-06-11T17:23:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:46.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:23:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672980 2024-06-11T17:23:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672980 2024-06-11T17:23:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-06-11T17:23:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:23:46.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:23:46.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-06-11T17:23:46.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-06-11T17:23:46.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509453' 2024-06-11T17:23:46.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:46.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182402 2024-06-11T17:23:46.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:46.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:23:46.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182402 2024-06-11T17:23:46.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:46.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182402 2024-06-11T17:23:46.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-11T17:23:46.898 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 107374182402 2024-06-11T17:23:46.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:23:47.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-11T17:23:47.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:47.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672980 2024-06-11T17:23:47.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:47.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:23:47.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-06-11T17:23:47.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:47.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672980 2024-06-11T17:23:47.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-11T17:23:47.264 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672980 2024-06-11T17:23:47.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:23:47.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-11T17:23:47.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:23:47.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-06-11T17:23:47.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:23:47.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:23:47.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-06-11T17:23:47.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:23:47.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-06-11T17:23:47.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-11T17:23:47.616 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509453 2024-06-11T17:23:47.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:23:47.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-11T17:23:47.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:23:47.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:23:47.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:23:48.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:23:48.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:23:48.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:23:48.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:23:48.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:23:48.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:23:48.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:23:48.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:23:48.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:23:48.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:23:48.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:23:48.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:23:49.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:23:49.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:23:49.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:23:49.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-06-11T17:23:49.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T17:23:49.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:23:49.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-11T17:23:49.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:23:49.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-06-11T17:23:49.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-06-11T17:23:49.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:23:49.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:23:49.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:23:49.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:23:49.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-06-11T17:23:49.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-06-11T17:23:49.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:49.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:49.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:49.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:50.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:50.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-06-11T17:23:50.413 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T17:23:50.413 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": false, 2024-06-11T17:23:50.413 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T17:23:50.413 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-06-04T17:22:10.416438+0000" 2024-06-11T17:23:50.414 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T17:23:50.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:50.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T17:23:50.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:50.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:23:50.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:23:50.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:23:50.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:23:50.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:50.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:50.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:50.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:50.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:23:31.640632+0000 '>' 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:50.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:23:51.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:23:51.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:23:51.755 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:23:51.755 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:51.755 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:51.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:51.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:52.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:23:31.640632+0000 '>' 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:52.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:23:53.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:23:53.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:23:53.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:23:53.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:53.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:53.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:53.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:53.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:23:31.640632+0000 '>' 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:53.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:23:54.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:23:54.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:23:54.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:23:54.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:54.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:54.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:54.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:54.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:23:31.640632+0000 '>' 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:54.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:23:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:23:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:23:55.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T17:23:55.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:23:55.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:23:55.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:23:55.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:23:56.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:23:52.186432+0000 '>' 2024-06-11T17:23:31.640632+0000 2024-06-11T17:23:56.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:23:56.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-11T17:23:56.395 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:23:56.395 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:23:52.475482+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-06-11T17:23:52.186432+0000 21'1 2024-06-11T17:23:52.186432+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:23:52.186432+0000 1 0 2024-06-11T17:23:56.395 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:23:56.395 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:23:56.396 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:23:56.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-11T17:23:58.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-11T17:23:58.729 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:23:58.730 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:23:52.475482+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-06-11T17:23:52.186432+0000 21'1 2024-06-11T17:23:52.186432+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:23:52.186432+0000 1 0 2024-06-11T17:23:58.730 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:23:58.730 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:23:58.730 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:23:58.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-11T17:24:00.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-11T17:24:01.061 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:24:01.061 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:23:52.475482+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-06-11T17:23:52.186432+0000 21'1 2024-06-11T17:23:52.186432+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:23:52.186432+0000 1 0 2024-06-11T17:24:01.062 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:24:01.062 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:24:01.062 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:24:01.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-06-11T17:24:06.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-11T17:24:06.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:24:06.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:24:06.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:24:06.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:24:06.083 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:24:06.083 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:24:06.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:24:06.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:24:06.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:24:06.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:06.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:24:06.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182407 2024-06-11T17:24:06.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182407 2024-06-11T17:24:06.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182407' 2024-06-11T17:24:06.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:06.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:24:06.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-11T17:24:06.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-11T17:24:06.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182407 1-42949672985' 2024-06-11T17:24:06.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:06.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:24:07.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-06-11T17:24:07.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-06-11T17:24:07.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-107374182407 1-42949672985 2-64424509458' 2024-06-11T17:24:07.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:07.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:07.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-107374182407 2024-06-11T17:24:07.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:24:07.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-107374182407 2024-06-11T17:24:07.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:07.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182407 2024-06-11T17:24:07.026 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 107374182407 2024-06-11T17:24:07.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-06-11T17:24:07.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:24:07.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182407 -lt 107374182407 2024-06-11T17:24:07.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:07.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-11T17:24:07.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:07.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:24:07.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-11T17:24:07.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:07.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-11T17:24:07.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-11T17:24:07.374 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672985 2024-06-11T17:24:07.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:24:07.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-06-11T17:24:07.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:07.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-06-11T17:24:07.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:07.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:24:07.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-06-11T17:24:07.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:07.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-06-11T17:24:07.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-11T17:24:07.704 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509458 2024-06-11T17:24:07.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:24:08.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-11T17:24:08.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:24:08.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:24:08.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:24:08.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:24:08.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:24:08.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:24:08.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:24:08.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:24:08.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:24:08.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:24:08.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:24:08.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:24:08.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:24:08.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:24:08.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:24:09.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:24:09.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:24:09.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:24:09.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-11T17:24:09.525 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:24:09.526 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:23:52.475482+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-06-11T17:23:52.186432+0000 21'1 2024-06-11T17:23:52.186432+0000 0 0 periodic scrub scheduled @ 2024-06-12T17:23:52.186432+0000 1 0 2024-06-11T17:24:09.526 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:24:09.526 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:24:09.526 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:24:09.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-11T17:24:09.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:24:09.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:24:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:24:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:24:09.546 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:24:09.546 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:24:09.546 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:24:09.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:24:09.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:24:09.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:24:10.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:24:10.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:24:10.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:24:10.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:24:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:24:10.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-11T17:24:10.786 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:24:10.786 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:24:11.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:24:11.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:11.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-11T17:24:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:24:11.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:24:11.086 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:24:11.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-11T17:24:11.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:24:11.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:24:11.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:24:11.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:24:11.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:24:11.124 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:11.123+0000 7f089ab127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:11.219 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:11.219+0000 7f089ab127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:11.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:11.219+0000 7f089ab127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:11.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:11.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:11.764 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:11.763+0000 7f089ab127c0 -1 Falling back to public interface 2024-06-11T17:24:12.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:12.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:12.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:24:12.764 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:24:12.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:12.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:13.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:14.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:14.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:14.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:24:14.098 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:24:14.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:14.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:14.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:14.760 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:14.759+0000 7f089ab127c0 -1 osd.0 26 log_to_monitors true 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/444401553,v1:127.0.0.1:6803/444401553] [v2:127.0.0.1:6804/444401553,v1:127.0.0.1:6805/444401553] exists,up 5e07f7e6-5f12-4c91-afaf-f8d057014c5c 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:24:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:24:15.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:24:15.812 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:24:15.812 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:24:15.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:24:15.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:24:15.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:24:15.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:24:15.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:24:15.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:24:15.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:24:15.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:24:15.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:16.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:24:16.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T17:24:16.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T17:24:16.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882' 2024-06-11T17:24:16.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:16.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:24:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-06-11T17:24:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-06-11T17:24:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672988' 2024-06-11T17:24:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:24:16.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:24:16.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509461 2024-06-11T17:24:16.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509461 2024-06-11T17:24:16.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-42949672988 2-64424509461' 2024-06-11T17:24:16.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:16.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018882 2024-06-11T17:24:16.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:16.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:24:16.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018882 2024-06-11T17:24:16.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:16.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T17:24:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-11T17:24:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 128849018882 2024-06-11T17:24:16.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:24:17.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 128849018882 2024-06-11T17:24:17.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:24:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:24:18.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:24:18.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T17:24:18.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:18.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-06-11T17:24:18.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:18.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:24:18.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-06-11T17:24:18.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:18.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-06-11T17:24:18.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-06-11T17:24:18.469 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672988 2024-06-11T17:24:18.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:24:18.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-06-11T17:24:18.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:24:18.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509461 2024-06-11T17:24:18.814 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:24:18.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:24:18.816 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509461 2024-06-11T17:24:18.816 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:24:18.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509461 2024-06-11T17:24:18.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-11T17:24:18.817 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509461 2024-06-11T17:24:18.817 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:24:19.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-11T17:24:19.151 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:24:19.151 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:24:19.151 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:24:19.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:24:19.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:24:19.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:24:19.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:24:19.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:24:19.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:24:19.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:24:20.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:24:20.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:24:20.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:24:20.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:24:20.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:24:20.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:24:20.348 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:51.227+0000 7fb36ec05640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:51.227+0000 7fb36ec05640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:51.227+0000 7fb36ec05640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:52.179+0000 7fb36ec05640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:52.179+0000 7fb36ec05640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:52.179+0000 7fb36ec05640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:52.179+0000 7fb36ec05640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:23:52.179+0000 7fb36ec05640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:24:20.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-06-11T17:24:20.350 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-06-11T17:24:20.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:24:20.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:24:20.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:24:20.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:24:20.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:24:20.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:24:20.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:24:20.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:24:20.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:24:20.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:24:20.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:24:20.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:24:20.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:24:20.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:24:20.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:24:20.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:24:20.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:24:20.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:20.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:20.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:24:20.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:24:20.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:24:20.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:24:20.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:24:20.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:24:20.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:24:20.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:24:20.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:24:20.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:24:20.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:24:20.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:24:20.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:24:20.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:24:20.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:24:20.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:24:20.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:24:20.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:24:20.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:24:20.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:20.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:20.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:24:20.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:24:20.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:24:20.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:24:20.846 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:24:20.846 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:20.846 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:20.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:24:20.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:24:20.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:24:20.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:24:21.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:24:21.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:21.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:21.341 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:21.341 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:21.341 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:21.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:21.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:24:21.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:24:21.563 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:21.566 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:21.566 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:24:21.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:24:21.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:24:21.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:24:21.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:24:21.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:24:21.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:24:21.683 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:24:21.683 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:24:21.683 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:24:21.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:24:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:24:22.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:24:22.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:22.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:22.662 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:22.662 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:22.662 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:22.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:22.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:24:22.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:24:22.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:24:22.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-06-11T17:24:22.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:24:22.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:22.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:24:22.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:24:22.708 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:24:22.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:24:22.708 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:24:22.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dd22d598-298e-41fd-8bc5-fd40262a889d' 2024-06-11T17:24:22.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:24:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGiGhmfcZWKxAA+vdc7mmsVs8u3+NWSlmjtQ== 2024-06-11T17:24:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGiGhmfcZWKxAA+vdc7mmsVs8u3+NWSlmjtQ=="}' 2024-06-11T17:24:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd22d598-298e-41fd-8bc5-fd40262a889d -i td/osd-scrub-repair/0/new.json 2024-06-11T17:24:22.962 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:24:22.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:24:22.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBGiGhmfcZWKxAA+vdc7mmsVs8u3+NWSlmjtQ== --osd-uuid dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:24:23.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.011+0000 7f574efc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:23.014 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.011+0000 7f574efc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:23.015 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.015+0000 7f574efc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.015+0000 7f574efc47c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:24:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.015+0000 7f574efc47c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:24:23.016 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.015+0000 7f574efc47c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:24:23.104 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:23.103+0000 7f574efc47c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:24:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:24:29.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:24:29.576 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:24:29.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:24:29.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:24:30.004 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:24:30.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:24:30.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:30.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:24:30.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:24:30.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:24:30.041 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:30.039+0000 7fe92ef127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:30.068 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:30.067+0000 7fe92ef127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:30.069 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:30.067+0000 7fe92ef127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:30.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:24:30.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:30.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:30.625 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:30.623+0000 7fe92ef127c0 -1 Falling back to public interface 2024-06-11T17:24:30.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:31.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:31.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:31.712 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:24:31.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:24:31.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:31.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:32.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:33.047 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:24:33.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:24:33.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:33.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:33.260 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:33.259+0000 7fe92ef127c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:24:33.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:34.385 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:24:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:24:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:24:34.755 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/379565435,v1:127.0.0.1:6803/379565435] [v2:127.0.0.1:6804/379565435,v1:127.0.0.1:6805/379565435] exists,up dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:24:34.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:24:34.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:24:34.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:24:34.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:24:34.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:24:34.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:24:34.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:24:34.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:24:34.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03e65128-2f34-4ad1-b3d4-2f01a4141086 2024-06-11T17:24:34.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 03e65128-2f34-4ad1-b3d4-2f01a4141086' 2024-06-11T17:24:34.762 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 03e65128-2f34-4ad1-b3d4-2f01a4141086 2024-06-11T17:24:34.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:24:34.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSiGhmzhujLhAAVR83ZWwu9zrYeqCAPOH4FA== 2024-06-11T17:24:34.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSiGhmzhujLhAAVR83ZWwu9zrYeqCAPOH4FA=="}' 2024-06-11T17:24:34.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03e65128-2f34-4ad1-b3d4-2f01a4141086 -i td/osd-scrub-repair/1/new.json 2024-06-11T17:24:35.155 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:24:35.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:24:35.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBSiGhmzhujLhAAVR83ZWwu9zrYeqCAPOH4FA== --osd-uuid 03e65128-2f34-4ad1-b3d4-2f01a4141086 2024-06-11T17:24:35.216 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.215+0000 7f53213f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:35.248 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.247+0000 7f53213f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:35.250 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.247+0000 7f53213f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:35.251 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.247+0000 7f53213f57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:24:35.251 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.247+0000 7f53213f57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:24:35.251 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.247+0000 7f53213f57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:24:35.318 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:35.319+0000 7f53213f57c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:24:42.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:24:42.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:24:42.302 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:24:42.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:24:42.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:24:42.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:24:42.722 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:24:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:24:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:24:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:24:42.759 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:42.759+0000 7f70c439e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:42.787 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:42.787+0000 7f70c439e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:42.789 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:42.787+0000 7f70c439e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:43.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:24:43.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:24:43.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:24:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:24:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:24:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:43.069 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:24:43.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:24:43.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:43.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:24:43.365 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:43.363+0000 7f70c439e7c0 -1 Falling back to public interface 2024-06-11T17:24:43.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:44.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:44.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:44.449 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:24:44.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:24:44.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:44.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:24:44.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:45.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:45.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:45.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:24:45.788 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:24:45.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:45.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:24:46.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:46.192 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:46.191+0000 7f70c439e7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:24:47.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:47.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:47.129 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:24:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:24:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:24:47.508 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3413748942,v1:127.0.0.1:6811/3413748942] [v2:127.0.0.1:6812/3413748942,v1:127.0.0.1:6813/3413748942] exists,up 03e65128-2f34-4ad1-b3d4-2f01a4141086 2024-06-11T17:24:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:24:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:24:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:24:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:24:47.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:24:47.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:24:47.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:24:47.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03bfa7b5-cc95-419a-92cb-2424ac01c691 2024-06-11T17:24:47.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 03bfa7b5-cc95-419a-92cb-2424ac01c691' 2024-06-11T17:24:47.516 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 03bfa7b5-cc95-419a-92cb-2424ac01c691 2024-06-11T17:24:47.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:24:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfiGhmJtzPHxAAZDgXin4VEz79zgkCWFEWsQ== 2024-06-11T17:24:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfiGhmJtzPHxAAZDgXin4VEz79zgkCWFEWsQ=="}' 2024-06-11T17:24:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03bfa7b5-cc95-419a-92cb-2424ac01c691 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:24:47.966 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:24:47.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:24:47.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBfiGhmJtzPHxAAZDgXin4VEz79zgkCWFEWsQ== --osd-uuid 03bfa7b5-cc95-419a-92cb-2424ac01c691 2024-06-11T17:24:48.025 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.023+0000 7f51aef087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:48.060 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.059+0000 7f51aef087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:48.061 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.059+0000 7f51aef087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:48.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.059+0000 7f51aef087c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:24:48.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.059+0000 7f51aef087c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:24:48.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.059+0000 7f51aef087c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:24:48.144 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:48.143+0000 7f51aef087c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:24:54.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:24:54.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:24:54.894 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:24:54.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:24:54.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:24:55.317 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:24:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:24:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:24:55.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:24:55.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:24:55.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:24:55.354 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:55.351+0000 7fb2696437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:55.381 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:55.379+0000 7fb2696437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:55.382 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:55.379+0000 7fb2696437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:24:55.701 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:24:55.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:55.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:24:55.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:55.939+0000 7fb2696437c0 -1 Falling back to public interface 2024-06-11T17:24:56.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:57.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:57.072 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:24:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:24:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:57.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:24:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:58.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:58.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:58.401 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:24:58.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:24:58.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:58.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:24:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:24:58.647+0000 7fb2696437c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:24:58.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:24:59.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:24:59.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:24:59.771 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:24:59.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:24:59.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:24:59.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:25:00.107 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1857837780,v1:127.0.0.1:6819/1857837780] [v2:127.0.0.1:6820/1857837780,v1:127.0.0.1:6821/1857837780] exists,up 03bfa7b5-cc95-419a-92cb-2424ac01c691 2024-06-11T17:25:00.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:25:00.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:25:00.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:25:00.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-06-11T17:25:00.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-11T17:25:00.550 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T17:25:00.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:25:01.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-06-11T17:25:02.259 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:25:02.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:25:02.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:25:02.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:25:02.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:25:02.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:25:02.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:25:02.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:02.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:02.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:02.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:02.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-06-11T17:25:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-06-11T17:25:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-06-11T17:25:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:02.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:03.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672965 2024-06-11T17:25:03.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672965 2024-06-11T17:25:03.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-06-11T17:25:03.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:03.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:03.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509442 2024-06-11T17:25:03.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509442 2024-06-11T17:25:03.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-06-11T17:25:03.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:03.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-06-11T17:25:03.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:03.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:03.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-06-11T17:25:03.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:03.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-06-11T17:25:03.341 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836487 2024-06-11T17:25:03.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-11T17:25:03.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:03.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836487 2024-06-11T17:25:03.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:25:04.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:25:04.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:04.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-11T17:25:04.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:04.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672965 2024-06-11T17:25:04.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:04.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:04.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672965 2024-06-11T17:25:04.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:04.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672965 2024-06-11T17:25:04.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-06-11T17:25:04.994 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672965 2024-06-11T17:25:04.994 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672965 -lt 42949672965 2024-06-11T17:25:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:05.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:05.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509442 2024-06-11T17:25:05.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:05.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509442 2024-06-11T17:25:05.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:05.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509442 2024-06-11T17:25:05.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-06-11T17:25:05.341 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509442 2024-06-11T17:25:05.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:05.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509442 -lt 64424509442 2024-06-11T17:25:05.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:25:05.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:05.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:06.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:25:06.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:06.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:06.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:06.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:06.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:06.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:06.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:06.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:25:06.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:06.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:06.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:25:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-11T17:25:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:25:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-11T17:25:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:25:06.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:25:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:25:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:06.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:06.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:06.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:06.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:06.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:06.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:07.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:25:07.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:07.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:07.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:07.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:25:07.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:25:07.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:25:08.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:25:08.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:25:08.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:25:08.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:25:08.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:25:08.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:25:08.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:08.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:08.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:08.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:08.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:08.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:08.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:08.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T17:25:08.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:08.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:08.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:25:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:25:09.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:25:09.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:25:09.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:25:09.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:25:09.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T17:25:09.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:25:09.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T17:25:09.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:25:09.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:09.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:09.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:09.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:09.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:09.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:09.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:10.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:25:10.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:10.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:10.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:10.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:25:10.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:25:10.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:25:10.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-06-11T17:25:10.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-06-11T17:25:10.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:25:10.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-11T17:25:10.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:25:10.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:25:10.738 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:25:10.738 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:25:10.738 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:25:10.738 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:10.738 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:25:11.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:25:11.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:25:11.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:25:11.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:25:11.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:25:11.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:25:11.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T17:25:11.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:25:11.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:25:11.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:25:11.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:25:11.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:11.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-11T17:25:13.236 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T17:25:13.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:25:13.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:13.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:25:13.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:25:13.780 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:25:13.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:13.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:25:13.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:25:13.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:25:13.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:25:13.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:25:13.819 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:13.819+0000 7fc0bcca87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:13.854 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:13.851+0000 7fc0bcca87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:13.856 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:13.855+0000 7fc0bcca87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:25:14.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:14.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:14.382 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:14.379+0000 7fc0bcca87c0 -1 Falling back to public interface 2024-06-11T17:25:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:15.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:15.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:15.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:25:15.473 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:15.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:15.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:25:16.813 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:25:16.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:16.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:17.074 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:17.071+0000 7fc0bcca87c0 -1 osd.0 20 log_to_monitors true 2024-06-11T17:25:17.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:18.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:18.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:18.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:25:18.154 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:25:18.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:18.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/4104831983,v1:127.0.0.1:6803/4104831983] [v2:127.0.0.1:6804/4104831983,v1:127.0.0.1:6805/4104831983] exists,up dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:25:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:25:18.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:25:18.555 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:25:18.555 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:25:18.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:25:18.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:25:18.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:18.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:18.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:18.987 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:18.988 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:18.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:18.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:18.988 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:19.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215106 2024-06-11T17:25:19.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215106 2024-06-11T17:25:19.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106' 2024-06-11T17:25:19.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:19.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:19.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-06-11T17:25:19.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-06-11T17:25:19.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106 1-42949672969' 2024-06-11T17:25:19.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:19.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:19.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-06-11T17:25:19.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-06-11T17:25:19.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106 1-42949672969 2-64424509446' 2024-06-11T17:25:19.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:19.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215106 2024-06-11T17:25:19.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:19.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:19.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215106 2024-06-11T17:25:19.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:19.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215106 2024-06-11T17:25:19.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-11T17:25:19.437 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 103079215106 2024-06-11T17:25:19.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:19.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-11T17:25:19.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:19.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-06-11T17:25:19.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:19.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:19.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-06-11T17:25:19.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:19.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-06-11T17:25:19.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-11T17:25:19.779 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672969 2024-06-11T17:25:19.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:20.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-06-11T17:25:20.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:20.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-06-11T17:25:20.118 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:20.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-06-11T17:25:20.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-06-11T17:25:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-06-11T17:25:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509446 2024-06-11T17:25:20.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:20.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509446 2024-06-11T17:25:20.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:25:20.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:20.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:20.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:25:20.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:20.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:20.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:20.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:20.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:20.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:20.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:21.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:25:21.243 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:21.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:21.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:21.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:25:21.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:25:21.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:25:21.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-06-11T17:25:21.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T17:25:21.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:25:21.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:25:21.631 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:21.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-06-11T17:25:21.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-06-11T17:25:21.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:25:21.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:25:21.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:21.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:25:22.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-06-11T17:25:22.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-06-11T17:25:22.310 INFO:tasks.workunit.client.0.smithi012.stdout:Affected PG 1.0 w/ primary 1 2024-06-11T17:25:22.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-06-11T17:25:22.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:25:22.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:25:22.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:25:22.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:25:22.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-06-11T17:25:00.498014+0000 2024-06-11T17:25:22.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-06-11T17:25:22.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-06-11T17:25:22.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-06-11T17:25:22.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-06-11T17:25:22.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T17:25:22.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:25:22.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:25:22.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T17:25:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-06-11T17:25:00.498014+0000 2024-06-11T17:25:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-06-11T17:25:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T17:25:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T17:25:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-11T17:25:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-06-11T17:25:22.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T17:25:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T17:25:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T17:25:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T17:25:23.000 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:23.000 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:23.000 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:23.000 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T17:25:23.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-06-11T17:25:23.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T17:25:23.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-06-11T17:25:23.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T17:25:23.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T17:25:23.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-06-11T17:25:23.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T17:25:23.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T17:25:23.121 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T17:25:23.121 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T17:25:23.121 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T17:25:23.121 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:23.121 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:23.122 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:23.122 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T17:25:23.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-06-11T17:25:23.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T17:25:23.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-06-11T17:25:23.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:23.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:23.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:23.576 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:23.576 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:23.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:23.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:23.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:23.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215108 2024-06-11T17:25:23.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215108 2024-06-11T17:25:23.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215108' 2024-06-11T17:25:23.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:23.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:23.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672971 2024-06-11T17:25:23.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672971 2024-06-11T17:25:23.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215108 1-42949672971' 2024-06-11T17:25:23.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:23.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:24.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-06-11T17:25:24.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-06-11T17:25:24.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215108 1-42949672971 2-64424509448' 2024-06-11T17:25:24.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215108 2024-06-11T17:25:24.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:24.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:24.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215108 2024-06-11T17:25:24.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:24.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215108 2024-06-11T17:25:24.081 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 103079215108 2024-06-11T17:25:24.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-06-11T17:25:24.082 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:24.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215107 -lt 103079215108 2024-06-11T17:25:24.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:25:25.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:25:25.402 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:25.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215108 -lt 103079215108 2024-06-11T17:25:25.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:25.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672971 2024-06-11T17:25:25.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:25.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:25.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672971 2024-06-11T17:25:25.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:25.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672971 2024-06-11T17:25:25.746 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672971 2024-06-11T17:25:25.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-06-11T17:25:25.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:26.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672971 -lt 42949672971 2024-06-11T17:25:26.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:26.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-06-11T17:25:26.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:26.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:26.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-06-11T17:25:26.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:26.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-06-11T17:25:26.119 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509448 2024-06-11T17:25:26.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-11T17:25:26.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:26.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509448 2024-06-11T17:25:26.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-06-11T17:25:26.453 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11 17:25:26.456781884+00:00 2024-06-11T17:25:26.454 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T17:25:26.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-06-11T17:25:26.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-06-11T17:25:26.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-11T17:25:26.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-11T17:25:26.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-11T17:25:26.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-11T17:25:26.730 INFO:tasks.workunit.client.0.smithi012.stdout:{"deep":true,"must":false,"stamp":"2024-05-28T17:23:46.568229+0000"}0 ) state now: "active+clean" 2024-06-11T17:25:26.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-06-11T17:25:26.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-11T17:25:26.731 INFO:tasks.workunit.client.0.smithi012.stdout:"active+clean" 2024-06-11T17:25:26.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-11T17:25:26.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-06-11T17:25:26.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-11T17:25:27.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-11T17:25:27.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-11T17:25:27.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-11T17:25:27.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-11T17:25:27.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-11T17:25:27.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-06-11T17:25:27.193 INFO:tasks.workunit.client.0.smithi012.stdout:1 ) state now: "active+clean" 2024-06-11T17:25:27.193 INFO:tasks.workunit.client.0.smithi012.stdout:"active+clean" 2024-06-11T17:25:27.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-11T17:25:27.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-11T17:25:27.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-06-11T17:25:27.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-11T17:25:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-11T17:25:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-11T17:25:27.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-11T17:25:27.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-11T17:25:27.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-11T17:25:27.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stdout:2 ) state now: "active+clean" 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stdout:"active+clean" 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-06-11T17:25:27.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-11T17:25:27.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-11T17:25:27.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-11T17:25:27.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-11T17:25:27.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-11T17:25:28.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.124 INFO:tasks.workunit.client.0.smithi012.stdout:3 ) state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:28.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-11T17:25:28.124 INFO:tasks.workunit.client.0.smithi012.stdout:found scrub 2024-06-11T17:25:28.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-06-11T17:25:28.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-06-11T17:25:28.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-06-11T17:25:28.125 INFO:tasks.workunit.client.0.smithi012.stdout:initiate_and_fetch_state ret: 0 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-06-11T17:25:28.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-06-11T17:25:28.128 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:28.128 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:28.128 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:28.128 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.td/osd-scrub-repair.asok 2024-06-11T17:25:28.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-06-11T17:25:28.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:28.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:28.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:28.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.397 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:28.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:28.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-06-11T17:25:28.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-11T17:25:28.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-11T17:25:28.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:28.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:28.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:28.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.861 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:28.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:28.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:28.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-06-11T17:25:28.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-11T17:25:29.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-11T17:25:29.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:29.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:29.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:29.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:29.325 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:29.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:29.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:29.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-06-11T17:25:29.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-11T17:25:29.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-11T17:25:29.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:29.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:29.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:29.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:29.813 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:29.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:29.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:29.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-06-11T17:25:29.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-11T17:25:30.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-11T17:25:30.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:30.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:30.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:30.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-11T17:25:30.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-11T17:25:30.279 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-11T17:25:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-06-11T17:25:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-06-11T17:25:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:30.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:30.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:30.606 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:30.606 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:30.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:30.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:30.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:30.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215110 2024-06-11T17:25:30.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215110 2024-06-11T17:25:30.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215110' 2024-06-11T17:25:30.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:30.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:30.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-06-11T17:25:30.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-06-11T17:25:30.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215110 1-42949672973' 2024-06-11T17:25:30.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:30.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:31.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-06-11T17:25:31.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-06-11T17:25:31.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215110 1-42949672973 2-64424509451' 2024-06-11T17:25:31.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:31.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:31.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215110 2024-06-11T17:25:31.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:31.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215110 2024-06-11T17:25:31.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:31.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215110 2024-06-11T17:25:31.063 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 103079215110 2024-06-11T17:25:31.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-06-11T17:25:31.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:31.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215109 -lt 103079215110 2024-06-11T17:25:31.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:25:32.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:25:32.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215110 -lt 103079215110 2024-06-11T17:25:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:32.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-06-11T17:25:32.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:32.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:32.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:32.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-06-11T17:25:32.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-06-11T17:25:32.730 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672973 2024-06-11T17:25:32.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-11T17:25:32.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:33.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-11T17:25:33.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:33.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-06-11T17:25:33.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:33.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:33.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:33.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-06-11T17:25:33.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-06-11T17:25:33.068 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509451 2024-06-11T17:25:33.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-11T17:25:33.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:33.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-06-11T17:25:33.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-11T17:25:33.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-11T17:25:33.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-11T17:25:33.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-11T17:25:33.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-11T17:25:33.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-06-11T17:25:33.886 INFO:tasks.workunit.client.0.smithi012.stdout:wait-scrub-end state now: "active+clean" 2024-06-11T17:25:33.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-06-11T17:25:33.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:33.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-06-11T17:25:33.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-11T17:25:33.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-06-11T17:25:33.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-06-11T17:25:34.184 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T17:25:34.185 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-11T17:25:31.420575+0000 20'1 25:56 [1,0] 1 [1,0] 1 20'1 2024-06-11T17:25:31.094614+0000 20'1 2024-06-11T17:25:31.094614+0000 0 3 periodic scrub scheduled @ 2024-06-12T17:25:31.094614+0000 1 0 2024-06-11T17:25:34.185 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T17:25:34.185 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T17:25:34.185 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T17:25:34.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-11T17:25:34.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:25:34.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:25:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:25:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:25:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:25:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:25:34.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:25:34.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:34.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:25:34.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:25:34.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:25:34.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:25:35.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:25:35.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-11T17:25:35.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:25:35.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:25:35.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:25:35.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:25:35.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:35.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-11T17:25:35.323 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:25:35.323 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:25:35.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:35.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:25:35.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:25:35.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:25:35.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:35.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:25:35.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:25:35.615 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:25:35.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:35.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:25:35.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:25:35.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:25:35.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:25:35.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:25:35.660 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:35.659+0000 7fe0534307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:35.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:35.775+0000 7fe0534307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:35.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:35.779+0000 7fe0534307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:25:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:25:35.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:35.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:36.313 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:36.311+0000 7fe0534307c0 -1 Falling back to public interface 2024-06-11T17:25:36.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:37.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:37.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:37.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:25:37.321 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:37.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:37.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:37.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:38.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:38.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:38.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:25:38.658 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:25:38.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:38.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:38.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:39.217 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:39.215+0000 7fe0534307c0 -1 osd.0 25 log_to_monitors true 2024-06-11T17:25:39.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:39.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:39.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:25:39.992 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:25:39.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:39.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/161961781,v1:127.0.0.1:6803/161961781] [v2:127.0.0.1:6804/161961781,v1:127.0.0.1:6805/161961781] exists,up dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:25:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:25:40.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:25:40.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:25:40.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:25:40.375 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:25:40.375 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:25:40.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:25:40.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:25:40.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:40.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:40.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:41.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051586 2024-06-11T17:25:41.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051586 2024-06-11T17:25:41.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051586' 2024-06-11T17:25:41.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:41.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-06-11T17:25:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-06-11T17:25:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051586 1-42949672976' 2024-06-11T17:25:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:41.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:41.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-06-11T17:25:41.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-06-11T17:25:41.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051586 1-42949672976 2-64424509454' 2024-06-11T17:25:41.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:41.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051586 2024-06-11T17:25:41.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:41.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:41.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051586 2024-06-11T17:25:41.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:41.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051586 2024-06-11T17:25:41.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-11T17:25:41.377 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 124554051586 2024-06-11T17:25:41.377 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:41.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051586 2024-06-11T17:25:41.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:25:42.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:25:42.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-11T17:25:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:43.038 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-06-11T17:25:43.038 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:43.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-06-11T17:25:43.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:43.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-06-11T17:25:43.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-11T17:25:43.042 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672976 2024-06-11T17:25:43.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:43.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-11T17:25:43.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:43.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-06-11T17:25:43.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:43.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:43.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-06-11T17:25:43.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:43.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-06-11T17:25:43.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-11T17:25:43.377 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509454 2024-06-11T17:25:43.377 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:43.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-11T17:25:43.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:25:43.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:43.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:44.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:25:44.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:44.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:44.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:25:44.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:44.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:44.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:25:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:25:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:25:44.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-11T17:25:44.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T17:25:44.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:25:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-11T17:25:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T17:25:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:25:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:25:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:45.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T17:25:45.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-11T17:25:45.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:25:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T17:25:45.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:25:45.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:25:45.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:25:45.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:25:45.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:25:45.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:25:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:46.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:46.304 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:46.304 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:46.304 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-11T17:25:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:25:46.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T17:25:46.306 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T17:25:46.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-11T17:25:46.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T17:25:46.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T17:25:46.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:25:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:25:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:25:46.344 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:46.343+0000 7f5fa63897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:46.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:46.347+0000 7f5fa63897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:46.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:46.347+0000 7f5fa63897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T17:25:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:25:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:25:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:25:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:25:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:25:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:25:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:46.893 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:46.891+0000 7f5fa63897c0 -1 Falling back to public interface 2024-06-11T17:25:47.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:48.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:48.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:48.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:25:48.006 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:48.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:48.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:48.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:49.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:49.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:49.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:25:49.342 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:25:49.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:49.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:49.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:25:49.807 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:49.803+0000 7f5fa63897c0 -1 osd.0 30 log_to_monitors true 2024-06-11T17:25:50.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:25:50.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:25:50.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:25:50.681 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:25:50.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:25:50.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:25:51.160 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/2301330503,v1:127.0.0.1:6803/2301330503] [v2:127.0.0.1:6804/2301330503,v1:127.0.0.1:6805/2301330503] exists,up dd22d598-298e-41fd-8bc5-fd40262a889d 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:25:51.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:25:51.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:25:51.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:25:51.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:25:51.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:25:51.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:25:51.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:25:51.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:25:51.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:25:51.596 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:25:51.596 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:25:51.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:25:51.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:51.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:25:51.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888066 2024-06-11T17:25:51.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888066 2024-06-11T17:25:51.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-146028888066' 2024-06-11T17:25:51.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:51.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:25:51.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-06-11T17:25:51.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-06-11T17:25:51.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-146028888066 1-42949672979' 2024-06-11T17:25:51.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:25:51.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:25:52.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-06-11T17:25:52.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-06-11T17:25:52.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-146028888066 1-42949672979 2-64424509457' 2024-06-11T17:25:52.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:52.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-146028888066 2024-06-11T17:25:52.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:52.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:25:52.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-146028888066 2024-06-11T17:25:52.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:52.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888066 2024-06-11T17:25:52.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-06-11T17:25:52.113 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 146028888066 2024-06-11T17:25:52.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:52.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 146028888066 2024-06-11T17:25:52.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:25:53.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:25:53.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:25:53.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888066 -lt 146028888066 2024-06-11T17:25:53.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:53.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:53.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-06-11T17:25:53.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:25:53.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-06-11T17:25:53.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:53.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-06-11T17:25:53.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-11T17:25:53.767 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672979 2024-06-11T17:25:53.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:25:54.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672979 2024-06-11T17:25:54.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:25:54.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-06-11T17:25:54.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:25:54.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:25:54.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-06-11T17:25:54.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:25:54.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-06-11T17:25:54.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-11T17:25:54.108 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509457 2024-06-11T17:25:54.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:25:54.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-11T17:25:54.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:25:54.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:54.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:54.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T17:25:54.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:25:54.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:25:55.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T17:25:55.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:25:55.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:25:55.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:25:55.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T17:25:55.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:25:55.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:25:55.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:25:55.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-11T17:25:55.622 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:25:31.087+0000 7f70aab2d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-11T17:25:55.622 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:25:31.087+0000 7f70aab2d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-11T17:25:55.623 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:25:31.087+0000 7f70aab2d640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-11T17:25:55.623 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:25:31.087+0000 7f70aab2d640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-11T17:25:55.623 INFO:tasks.workunit.client.0.smithi012.stdout:2024-06-11T17:25:31.087+0000 7f70aab2d640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-11T17:25:55.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:25:55.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:25:55.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:25:55.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:25:55.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:25:55.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:25:55.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:25:55.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:25:55.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:25:55.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:25:55.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:25:55.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:25:55.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:25:55.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:25:55.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:25:55.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:25:55.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:25:55.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:55.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:55.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:25:55.750 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:25:55.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:25:55.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:25:55.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:25:55.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:25:55.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:25:55.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:25:55.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:25:55.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:25:55.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:25:55.755 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:25:55.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:25:55.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:25:55.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:25:55.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:25:55.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:25:55.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:25:55.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:25:55.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:25:55.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:55.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:25:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:25:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:25:55.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:25:55.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:25:55.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:55.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:55.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:25:55.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:25:55.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:25:55.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:25:55.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:25:55.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:25:56.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:25:56.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:56.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:56.278 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:56.278 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:56.278 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:56.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:56.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:25:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:25:56.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:25:56.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:25:56.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:25:56.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:25:56.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:25:56.483 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:25:56.483 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:25:56.483 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:25:56.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:25:56.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:25:56.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:25:56.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:25:56.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:25:56.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:25:56.605 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:25:56.605 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:25:56.605 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:25:56.606 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:25:56.606 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:56.606 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:56.606 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:25:56.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:25:56.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:25:56.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:25:56.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:57.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:57.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:25:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:25:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:25:57.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:25:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:25:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:25:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:25:57.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:25:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6585fc5c-b030-4348-8bca-6fa74be2fd13 2024-06-11T17:25:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6585fc5c-b030-4348-8bca-6fa74be2fd13' 2024-06-11T17:25:57.634 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 6585fc5c-b030-4348-8bca-6fa74be2fd13 2024-06-11T17:25:57.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:25:57.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCliGhmsa70JhAASCGKWB7Hc7CJuaKyHaERYg== 2024-06-11T17:25:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCliGhmsa70JhAASCGKWB7Hc7CJuaKyHaERYg=="}' 2024-06-11T17:25:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6585fc5c-b030-4348-8bca-6fa74be2fd13 -i td/osd-scrub-repair/0/new.json 2024-06-11T17:25:57.891 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:25:57.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:25:57.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCliGhmsa70JhAASCGKWB7Hc7CJuaKyHaERYg== --osd-uuid 6585fc5c-b030-4348-8bca-6fa74be2fd13 2024-06-11T17:25:57.950 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.946+0000 7f411cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:57.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.950+0000 7f411cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:57.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.950+0000 7f411cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:25:57.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.950+0000 7f411cbfc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:25:57.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.950+0000 7f411cbfc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:25:57.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:57.950+0000 7f411cbfc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:25:58.033 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:25:58.030+0000 7f411cbfc7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:26:04.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:26:04.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:26:04.475 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:26:04.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:26:04.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:26:04.886 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:26:04.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:26:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:26:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:26:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:26:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:26:04.923 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:04.918+0000 7f51237137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:04.925 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:04.922+0000 7f51237137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:04.926 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:04.922+0000 7f51237137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:26:05.209 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:26:05.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:05.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:05.482 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:05.478+0000 7f51237137c0 -1 Falling back to public interface 2024-06-11T17:26:05.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:06.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:06.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:06.565 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:26:06.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:26:06.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:06.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:06.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:07.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:07.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:07.895 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:26:07.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:26:07.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:07.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:08.156 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:08.154+0000 7f51237137c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:26:08.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:09.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:09.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:09.275 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:26:09.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:26:09.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:09.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:09.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:10.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:10.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:10.612 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:26:10.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:26:10.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:10.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:11.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:12.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:12.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:12.007 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:26:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:26:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:13.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:13.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:13.343 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:26:13.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:26:13.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:13.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:13.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:14.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:14.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:14.681 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:26:14.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:26:14.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:14.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:15.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:16.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:16.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:16.154 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:26:16.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:26:16.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:16.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:16.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:17.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:17.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:17.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:26:17.484 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:26:17.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:17.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:17.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:18.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:18.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:18.818 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:26:18.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:26:18.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:18.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:20.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:26:21.534 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:26:21.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:21.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:21.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:22.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:22.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:22.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:26:22.874 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:26:22.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:22.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:23.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:24.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:24.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:24.559 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:26:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:26:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:25.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:26.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:26.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:26.050 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:26:26.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:26:26.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:26.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:26.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:27.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:27.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:27.613 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:26:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:26:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:28.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:29.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:29.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:29.389 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:26:29.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:26:29.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:29.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:30.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:31.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:31.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:31.112 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:26:31.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:26:31.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:31.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:32.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:32.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:32.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:26:32.920 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:26:32.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:32.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:26:33.280 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/178410462,v1:127.0.0.1:6803/178410462] [v2:127.0.0.1:6804/178410462,v1:127.0.0.1:6805/178410462] exists,up 6585fc5c-b030-4348-8bca-6fa74be2fd13 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:26:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:26:33.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:26:33.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:26:33.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:26:33.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:26:33.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:26:33.288 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:26:33.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6f8fba44-39d8-4090-aa84-40630a235c0c' 2024-06-11T17:26:33.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:26:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJiGhmsnBrEhAAHycJj1iyDMPaiZ/CNUrPVg== 2024-06-11T17:26:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJiGhmsnBrEhAAHycJj1iyDMPaiZ/CNUrPVg=="}' 2024-06-11T17:26:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f8fba44-39d8-4090-aa84-40630a235c0c -i td/osd-scrub-repair/1/new.json 2024-06-11T17:26:33.702 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:26:33.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:26:33.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDJiGhmsnBrEhAAHycJj1iyDMPaiZ/CNUrPVg== --osd-uuid 6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:26:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.766+0000 7f76a8aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:33.795 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.794+0000 7f76a8aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:33.797 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.798+0000 7f76a8aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:33.797 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.798+0000 7f76a8aec7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:26:33.797 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.798+0000 7f76a8aec7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:26:33.797 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.798+0000 7f76a8aec7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:26:33.868 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:33.870+0000 7f76a8aec7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:26:40.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:26:40.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:26:40.753 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:26:40.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:26:40.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:26:41.178 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:26:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:26:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:26:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:26:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:26:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:26:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:41.214+0000 7fbbd59097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:41.236 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:41.238+0000 7fbbd59097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:41.238+0000 7fbbd59097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:26:41.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:26:41.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:26:41.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:26:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:41.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:41.786 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:41.786+0000 7fbbd59097c0 -1 Falling back to public interface 2024-06-11T17:26:41.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:42.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:42.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:42.851 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:26:42.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:26:42.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:42.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:43.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:44.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:44.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:44.187 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:26:44.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:26:44.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:44.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:44.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:44.692 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:26:44.694+0000 7fbbd59097c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:26:45.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:45.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:45.515 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:26:45.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:26:45.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:45.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:45.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:46.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:46.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:46.918 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:26:46.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:26:46.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:46.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:47.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:48.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:48.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:48.387 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:26:48.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:26:48.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:48.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:48.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:49.841 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:26:49.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:26:49.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:49.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:50.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:51.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:51.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:51.272 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:26:51.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:26:51.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:51.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:51.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:52.731 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:26:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:26:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:53.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:54.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:54.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:54.169 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:26:54.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:26:54.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:54.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:54.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:55.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:55.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:55.507 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:26:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:26:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:56.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:57.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:57.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:57.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:26:57.012 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:26:57.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:57.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:58.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:58.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:58.490 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:26:58.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:26:58.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:58.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:26:58.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:26:59.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:26:59.996 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:26:59.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:26:59.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:26:59.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:26:59.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:00.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:01.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:01.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:01.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:27:01.677 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:27:01.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:01.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:02.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:03.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:03.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:03.597 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:27:03.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:27:03.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:03.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:04.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:05.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:05.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:05.490 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:27:05.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:27:05.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:05.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:06.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:27:07.197 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:27:07.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:07.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:07.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:08.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:08.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:08.712 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:27:08.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:27:08.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:08.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:09.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:10.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:10.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:10.601 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:27:10.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:27:10.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:10.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:27:10.965 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2472988692,v1:127.0.0.1:6811/2472988692] [v2:127.0.0.1:6812/2472988692,v1:127.0.0.1:6813/2472988692] exists,up 6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:27:10.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:27:10.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:27:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:27:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:27:10.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:27:10.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:27:10.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae70aae4-7ed2-4676-857d-44b5d6a34a6c 2024-06-11T17:27:10.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ae70aae4-7ed2-4676-857d-44b5d6a34a6c' 2024-06-11T17:27:10.972 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 ae70aae4-7ed2-4676-857d-44b5d6a34a6c 2024-06-11T17:27:10.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:27:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuiGhm0weFOxAAUYSy+vBdjcMe4gECA0K4GQ== 2024-06-11T17:27:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuiGhm0weFOxAAUYSy+vBdjcMe4gECA0K4GQ=="}' 2024-06-11T17:27:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae70aae4-7ed2-4676-857d-44b5d6a34a6c -i td/osd-scrub-repair/2/new.json 2024-06-11T17:27:11.405 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:27:11.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:27:11.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDuiGhm0weFOxAAUYSy+vBdjcMe4gECA0K4GQ== --osd-uuid ae70aae4-7ed2-4676-857d-44b5d6a34a6c 2024-06-11T17:27:11.465 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.466+0000 7f63b01027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:11.489 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.490+0000 7f63b01027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:11.491 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.490+0000 7f63b01027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:11.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.494+0000 7f63b01027c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:27:11.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.494+0000 7f63b01027c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:27:11.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.494+0000 7f63b01027c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:27:11.559 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:11.558+0000 7f63b01027c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:27:18.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:27:18.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:27:18.575 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:27:18.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:27:18.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:27:19.014 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:27:19.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:27:19.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:27:19.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:27:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:27:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:27:19.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:19.050+0000 7efc36eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:19.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:19.086+0000 7efc36eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:19.086+0000 7efc36eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:27:19.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:27:19.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:27:19.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:19.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:19.633 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:19.634+0000 7efc36eeb7c0 -1 Falling back to public interface 2024-06-11T17:27:19.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:20.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:20.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:20.716 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:27:20.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:27:20.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:20.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:21.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:22.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:22.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:22.042 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:27:22.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:27:22.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:22.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:22.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:27:22.542+0000 7efc36eeb7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:27:23.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:23.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:23.499 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:27:23.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:27:23.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:23.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:23.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:24.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:24.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:24.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:27:24.969 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:27:24.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:24.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:25.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:27:26.343 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:27:26.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:26.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:26.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:27:27.780 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:27:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:28.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:29.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:29.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:29.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:27:29.369 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:27:29.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:29.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:29.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:30.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:30.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:30.804 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:27:30.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:27:30.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:30.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:31.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:32.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:32.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:32.371 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:27:32.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:27:32.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:32.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:32.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:33.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:33.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:33.860 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:27:33.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:27:33.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:33.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:34.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:35.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:35.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:35.205 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:27:35.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:27:35.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:35.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:35.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:36.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:36.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:36.541 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:27:36.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:27:36.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:36.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:36.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:37.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:37.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:37.871 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:27:37.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:27:37.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:37.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:38.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:39.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:39.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:39.419 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:27:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:27:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:39.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:40.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:40.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:40.991 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:27:40.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:27:40.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:40.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:41.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:42.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:42.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:42.839 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:27:42.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:27:42.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:42.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:44.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:44.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:44.437 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:27:44.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:27:44.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:44.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:44.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:45.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:45.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:45.914 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:27:45.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:27:45.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:45.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:46.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:47.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:47.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:47.627 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:27:47.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:27:47.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:47.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:47.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:48.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:48.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:48.966 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:27:48.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:27:48.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:48.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:49.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:27:50.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:27:50.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:27:50.439 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:27:50.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:27:50.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:27:50.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:27:50.787 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2475744152,v1:127.0.0.1:6819/2475744152] [v2:127.0.0.1:6820/2475744152,v1:127.0.0.1:6821/2475744152] exists,up ae70aae4-7ed2-4676-857d-44b5d6a34a6c 2024-06-11T17:27:50.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:27:50.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:27:50.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:27:50.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-11T17:27:50.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:27:51.086 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:27:51.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:27:51.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:27:51.532 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:27:51.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:27:52.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:27:55.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-11T17:27:55.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:27:55.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:27:55.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:27:55.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:27:55.086 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:27:55.086 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:27:55.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:27:55.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:27:55.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:27:55.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:27:55.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:27:55.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:27:55.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:27:55.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:27:55.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:27:55.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:27:55.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-06-11T17:27:55.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-06-11T17:27:55.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-06-11T17:27:55.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:27:55.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:27:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476750 2024-06-11T17:27:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476750 2024-06-11T17:27:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-68719476750' 2024-06-11T17:27:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:27:55.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:27:56.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888073 2024-06-11T17:27:56.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888073 2024-06-11T17:27:56.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-68719476750 2-146028888073' 2024-06-11T17:27:56.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:27:56.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-06-11T17:27:56.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:27:56.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:27:56.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-06-11T17:27:56.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:27:56.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-06-11T17:27:56.137 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836507 2024-06-11T17:27:56.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-11T17:27:56.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:27:56.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-11T17:27:56.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:27:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:27:57.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:27:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-11T17:27:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:27:57.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476750 2024-06-11T17:27:57.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:27:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:27:57.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:27:57.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476750 2024-06-11T17:27:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476750 2024-06-11T17:27:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-06-11T17:27:57.854 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 68719476750 2024-06-11T17:27:57.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:27:58.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476751 -lt 68719476750 2024-06-11T17:27:58.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:27:58.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:27:58.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888073 2024-06-11T17:27:58.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:27:58.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888073 2024-06-11T17:27:58.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:27:58.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888073 2024-06-11T17:27:58.180 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 146028888073 2024-06-11T17:27:58.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888073' 2024-06-11T17:27:58.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:27:58.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888073 -lt 146028888073 2024-06-11T17:27:58.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:27:58.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:27:58.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:27:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:27:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:27:59.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:27:59.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:27:59.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:27:59.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:27:59.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:27:59.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:27:59.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:27:59.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:27:59.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:27:59.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:27:59.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-11T17:28:00.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:28:00.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:28:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:28:01.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:28:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T17:28:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:28:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:28:02.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:28:02.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:28:02.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:28:02.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:28:02.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:28:02.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:28:02.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:28:02.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:28:02.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:28:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:28:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:28:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:28:03.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:28:03.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-06-11T17:28:03.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-06-11T17:28:03.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-06-11T17:28:03.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:03.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:28:03.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476753 2024-06-11T17:28:03.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476753 2024-06-11T17:28:03.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-68719476753' 2024-06-11T17:28:03.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:03.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:28:03.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888075 2024-06-11T17:28:03.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888075 2024-06-11T17:28:03.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-68719476753 2-146028888075' 2024-06-11T17:28:03.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:03.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-06-11T17:28:03.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:03.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:28:03.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-06-11T17:28:03.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:03.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-06-11T17:28:03.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-06-11T17:28:03.978 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836510 2024-06-11T17:28:03.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:04.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836510 2024-06-11T17:28:04.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:28:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:28:05.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:05.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-06-11T17:28:05.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:05.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:05.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476753 2024-06-11T17:28:05.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:28:05.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:05.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476753 2024-06-11T17:28:05.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476753 2024-06-11T17:28:05.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476753' 2024-06-11T17:28:05.643 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 68719476753 2024-06-11T17:28:05.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:28:06.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476753 -lt 68719476753 2024-06-11T17:28:06.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:06.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888075 2024-06-11T17:28:06.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:06.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:28:06.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888075 2024-06-11T17:28:06.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:06.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888075 2024-06-11T17:28:06.018 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 146028888075 2024-06-11T17:28:06.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888075' 2024-06-11T17:28:06.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:28:06.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888075 -lt 146028888075 2024-06-11T17:28:06.417 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:28:06.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:06.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:06.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:28:06.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:06.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:07.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:28:07.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:07.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:07.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:07.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:28:07.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:28:07.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:28:07.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:28:07.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-11T17:28:07.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-11T17:28:07.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:28:07.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-11T17:28:07.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-11T17:28:07.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:28:07.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-11T17:28:07.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:28:07.731 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:28:07.732 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:28:07.732 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:28:08.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-11T17:28:08.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:28:08.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-11T17:28:08.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:28:08.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:28:08.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:28:08.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:28:08.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:28:09.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:28:09.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:28:09.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:28:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:28:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:28:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:28:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:28:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:28:11.416 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T17:28:11.417 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-11T17:28:11.417 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:28:11.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:28:11.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:11.960 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:11.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:28:11.962 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:28:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:11.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:28:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:28:11.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:28:11.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:28:11.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:28:12.001 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:12.001+0000 7fae154c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:12.004 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:12.005+0000 7fae154c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:12.005 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:12.005+0000 7fae154c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:12.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:12.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:12.559 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:12.557+0000 7fae154c77c0 -1 Falling back to public interface 2024-06-11T17:28:12.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:13.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:13.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:13.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:28:13.677 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:13.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:13.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:14.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:15.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:15.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:15.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:28:15.010 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:28:15.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:15.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:15.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:15.660 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:15.661+0000 7fae154c77c0 -1 osd.1 58 log_to_monitors true 2024-06-11T17:28:16.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:16.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:16.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:28:16.339 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:28:16.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:16.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 64 up_thru 37 down_at 59 last_clean_interval [16,58) [v2:127.0.0.1:6810/2851295336,v1:127.0.0.1:6811/2851295336] [v2:127.0.0.1:6812/2851295336,v1:127.0.0.1:6813/2851295336] exists,up 6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:28:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:28:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:28:16.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:28:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:28:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:28:16.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:28:16.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:28:16.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:28:16.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:28:17.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:28:17.348 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:17.348 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:28:17.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:28:17.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:17.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:28:17.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-06-11T17:28:17.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-06-11T17:28:17.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-06-11T17:28:17.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:17.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:28:17.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906946 2024-06-11T17:28:17.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906946 2024-06-11T17:28:17.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-274877906946' 2024-06-11T17:28:17.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:17.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:28:18.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888079 2024-06-11T17:28:18.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888079 2024-06-11T17:28:18.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-274877906946 2-146028888079' 2024-06-11T17:28:18.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:18.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-06-11T17:28:18.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:18.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:28:18.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-06-11T17:28:18.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:18.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-06-11T17:28:18.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-11T17:28:18.081 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836513 2024-06-11T17:28:18.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:18.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836513 2024-06-11T17:28:18.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:28:19.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:28:19.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836513 2024-06-11T17:28:20.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:20.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-274877906946 2024-06-11T17:28:20.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:20.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:28:20.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-274877906946 2024-06-11T17:28:20.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:20.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906946 2024-06-11T17:28:20.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 274877906946' 2024-06-11T17:28:20.138 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 274877906946 2024-06-11T17:28:20.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:28:20.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906946 -lt 274877906946 2024-06-11T17:28:20.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:20.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888079 2024-06-11T17:28:20.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:20.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:28:20.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:20.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888079 2024-06-11T17:28:20.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888079 2024-06-11T17:28:20.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888079' 2024-06-11T17:28:20.464 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 146028888079 2024-06-11T17:28:20.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:28:20.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888080 -lt 146028888079 2024-06-11T17:28:20.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:28:20.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:20.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:21.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:28:21.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:21.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:21.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:28:21.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:21.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:21.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:21.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:28:21.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T17:28:21.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:28:21.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:28:21.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:28:21.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:28:22.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:28:22.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:22.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:22.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:28:22.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:22.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:22.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:28:22.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:28:22.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:28:22.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:28:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:28:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:28:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:28:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:28:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:28:23.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:28:23.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:23.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:23.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:23.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:23.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:23.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:23.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:23.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:28:23.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:23.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:23.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:28:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:28:24.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:28:24.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:28:24.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:28:24.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:28:24.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:28:24.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T17:28:24.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:28:24.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T17:28:24.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:28:24.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:24.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:24.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:24.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:24.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:24.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:24.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:25.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:28:25.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:25.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:25.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:25.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:28:25.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:28:25.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:28:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-11T17:28:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:28:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:28:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:28:25.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:28:26.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-11T17:28:26.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-11T17:28:26.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:28:26.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:28:26.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:28:26.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-11T17:28:08.586680+0000 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:28:08.586680+0000 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:28:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:28:26.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:28:26.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:28:26.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:28:26.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:28:26.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:28:26.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:28:24.528132+0000 '>' 2024-06-11T17:28:08.586680+0000 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:28:26.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:28:26.669 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:28:26.669 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:28:26.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:28:26.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:28:26.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:28:26.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:28:26.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:28:26.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:28:26.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:28:26.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:28:26.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:27.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:28:27.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-06-11T17:28:27.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-06-11T17:28:27.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-06-11T17:28:27.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:27.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:28:27.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906949 2024-06-11T17:28:27.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906949 2024-06-11T17:28:27.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-274877906949' 2024-06-11T17:28:27.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:27.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:28:27.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888082 2024-06-11T17:28:27.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888082 2024-06-11T17:28:27.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-274877906949 2-146028888082' 2024-06-11T17:28:27.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:27.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-06-11T17:28:27.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:27.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:28:27.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-06-11T17:28:27.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:27.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-06-11T17:28:27.551 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836516 2024-06-11T17:28:27.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-11T17:28:27.551 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:27.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836516 2024-06-11T17:28:27.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:28:28.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:28:28.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:29.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836517 -lt 21474836516 2024-06-11T17:28:29.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:29.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-274877906949 2024-06-11T17:28:29.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:29.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:28:29.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-274877906949 2024-06-11T17:28:29.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:29.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906949 2024-06-11T17:28:29.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 274877906949' 2024-06-11T17:28:29.284 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 274877906949 2024-06-11T17:28:29.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:28:29.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906949 -lt 274877906949 2024-06-11T17:28:29.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:29.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888082 2024-06-11T17:28:29.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:29.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:28:29.626 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888082 2024-06-11T17:28:29.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:29.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888082 2024-06-11T17:28:29.628 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 146028888082 2024-06-11T17:28:29.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888082' 2024-06-11T17:28:29.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:28:30.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888082 -lt 146028888082 2024-06-11T17:28:30.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:28:30.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:30.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:30.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:28:30.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:30.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:30.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:28:30.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:30.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:30.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:31.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:28:31.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:28:31.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:28:31.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-11T17:28:31.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-11T17:28:31.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:28:31.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-11T17:28:31.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:28:31.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:28:31.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:28:31.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:28:31.671 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-11T17:28:31.671 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:28:31.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-11T17:28:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:28:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:28:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:28:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:28:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:28:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:28:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:28:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:28:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:28:33.475 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T17:28:33.475 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-11T17:28:33.475 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:28:33.475 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:28:33.475 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:28:33.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:33.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:28:33.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:28:33.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:28:33.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:28:33.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:28:33.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:33.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:28:33.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:28:33.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:28:33.768 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:28:33.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:33.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:28:33.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:28:33.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:28:33.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:28:33.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:28:33.813 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:33.813+0000 7fb67a2647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:33.816 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:33.813+0000 7fb67a2647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:33.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:33.817+0000 7fb67a2647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:28:34.105 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:28:34.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:34.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:34.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:34.349+0000 7fb67a2647c0 -1 Falling back to public interface 2024-06-11T17:28:34.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:35.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:35.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:35.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:28:35.443 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:35.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:35.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:35.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:36.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:36.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:36.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:28:36.780 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:28:36.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:36.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:37.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:37.759 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:37.757+0000 7fb67a2647c0 -1 osd.1 74 log_to_monitors true 2024-06-11T17:28:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:28:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:28:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:38.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:28:40.141 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 75 last_clean_interval [64,74) [v2:127.0.0.1:6810/2534034221,v1:127.0.0.1:6811/2534034221] [v2:127.0.0.1:6812/2534034221,v1:127.0.0.1:6813/2534034221] exists,up 6f8fba44-39d8-4090-aa84-40630a235c0c 2024-06-11T17:28:40.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:28:40.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:28:40.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:28:40.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:28:40.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:28:40.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:28:40.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:28:40.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:28:40.143 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:28:40.143 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:28:40.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:28:40.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:28:40.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:28:40.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:28:40.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:28:40.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:28:40.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:28:40.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:28:40.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:40.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:28:40.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836520 2024-06-11T17:28:40.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836520 2024-06-11T17:28:40.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520' 2024-06-11T17:28:40.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:40.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:28:41.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=343597383682 2024-06-11T17:28:41.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 343597383682 2024-06-11T17:28:41.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-343597383682' 2024-06-11T17:28:41.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:28:41.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:28:41.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888086 2024-06-11T17:28:41.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888086 2024-06-11T17:28:41.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-343597383682 2-146028888086' 2024-06-11T17:28:41.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:41.219 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836520 2024-06-11T17:28:41.219 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:41.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:28:41.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836520 2024-06-11T17:28:41.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:41.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836520 2024-06-11T17:28:41.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-06-11T17:28:41.222 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836520 2024-06-11T17:28:41.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:41.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836519 -lt 21474836520 2024-06-11T17:28:41.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:28:42.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:28:42.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:28:42.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836520 -lt 21474836520 2024-06-11T17:28:42.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:42.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-343597383682 2024-06-11T17:28:42.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:42.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:28:42.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-343597383682 2024-06-11T17:28:42.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:42.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=343597383682 2024-06-11T17:28:42.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-06-11T17:28:42.917 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 343597383682 2024-06-11T17:28:42.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:28:43.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 343597383682 -lt 343597383682 2024-06-11T17:28:43.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:28:43.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:28:43.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888086 2024-06-11T17:28:43.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:28:43.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:28:43.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888086 2024-06-11T17:28:43.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888086 2024-06-11T17:28:43.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888086' 2024-06-11T17:28:43.280 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 146028888086 2024-06-11T17:28:43.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:28:43.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888086 -lt 146028888086 2024-06-11T17:28:43.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:28:43.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:43.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:44.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:28:44.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:28:44.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:28:44.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:28:44.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:28:44.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:28:44.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:28:44.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:28:44.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:28:44.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:28:44.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:28:44.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:28:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:28:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:28:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:28:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:28:44.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:28:44.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:28:44.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:28:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:28:45.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:28:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:28:45.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:28:45.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:28:45.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:28:45.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:28:45.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:28:45.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:28:45.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:28:45.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:28:45.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:28:45.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:28:45.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:28:45.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:28:45.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:45.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:45.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:28:45.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:28:45.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:28:45.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:28:45.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:28:45.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:28:45.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:28:45.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:28:45.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:28:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:28:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:28:45.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:28:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:28:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:28:45.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:28:45.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:28:45.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:28:45.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:28:45.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:28:45.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:28:45.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:45.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:45.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:28:45.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:28:45.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:28:45.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:28:45.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:28:45.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:45.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:45.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:28:45.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:28:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:28:45.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:28:45.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:28:45.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:28:45.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:28:45.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:28:45.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:28:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:28:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:45.631 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:28:45.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:28:45.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:28:45.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:28:45.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:28:45.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:28:45.816 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:28:45.816 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:28:45.816 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:28:45.816 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:28:45.817 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:28:45.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:28:46.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:28:46.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:28:46.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:28:46.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:28:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-11T17:28:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:28:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:28:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:28:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:28:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:28:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:28:46.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:28:46.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:28:46.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:28:46.934 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:28:46.934 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:46.935 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:46.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:46.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:28:46.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:28:46.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:28:46.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-11T17:28:46.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:28:46.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:46.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:28:46.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:28:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:28:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf70138b-1eed-47b0-a906-48883c6b49ff 2024-06-11T17:28:46.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cf70138b-1eed-47b0-a906-48883c6b49ff' 2024-06-11T17:28:46.981 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 cf70138b-1eed-47b0-a906-48883c6b49ff 2024-06-11T17:28:46.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:28:46.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOiWhmLbiROxAAgZvYA4PjnwDK2CyV7QZv4w== 2024-06-11T17:28:46.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOiWhmLbiROxAAgZvYA4PjnwDK2CyV7QZv4w=="}' 2024-06-11T17:28:46.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf70138b-1eed-47b0-a906-48883c6b49ff -i td/osd-scrub-repair/0/new.json 2024-06-11T17:28:47.294 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:28:47.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:28:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBOiWhmLbiROxAAgZvYA4PjnwDK2CyV7QZv4w== --osd-uuid cf70138b-1eed-47b0-a906-48883c6b49ff 2024-06-11T17:28:47.353 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.353+0000 7f913bf3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:47.360 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.361+0000 7f913bf3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:47.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.361+0000 7f913bf3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:47.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.361+0000 7f913bf3b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:28:47.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.361+0000 7f913bf3b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:28:47.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.361+0000 7f913bf3b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:28:47.461 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:47.461+0000 7f913bf3b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:28:54.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:28:54.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:28:54.121 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:28:54.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:28:54.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:28:54.528 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:28:54.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:28:54.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:28:54.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:28:54.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:28:54.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:28:54.565 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:54.565+0000 7fa13404a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:54.593 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:54.593+0000 7fa13404a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:54.594 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:54.593+0000 7fa13404a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:28:54.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:28:54.890 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:28:54.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:54.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:28:55.142 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:55.141+0000 7fa13404a7c0 -1 Falling back to public interface 2024-06-11T17:28:55.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:56.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:56.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:56.231 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:28:56.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:28:56.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:56.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:28:56.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:57.558 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:28:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:28:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:28:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:28:57.701+0000 7fa13404a7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:28:57.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:28:58.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:28:58.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:28:58.895 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:28:58.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:28:58.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:28:58.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:28:59.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:00.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:00.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:00.264 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:29:00.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:29:00.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:00.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:00.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:01.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:01.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:01.748 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:29:01.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:29:01.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:01.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:02.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:03.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:03.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:03.083 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:29:03.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:29:03.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:03.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:03.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:04.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:04.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:04.484 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:29:04.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:29:04.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:04.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:04.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:05.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:05.821 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:29:05.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:29:05.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:05.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:06.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:07.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:07.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:07.164 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:29:07.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:29:07.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:07.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:07.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:08.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:08.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:08.498 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:29:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:29:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:08.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:09.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:09.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:09.837 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:29:09.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:29:09.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:09.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:10.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:11.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:11.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:11.168 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:29:11.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:29:11.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:11.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:11.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:12.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:12.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:12.796 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:29:12.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:29:12.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:12.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:13.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:14.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:14.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:14.623 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:29:14.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:29:14.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:14.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:15.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:16.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:16.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:16.297 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:29:16.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:29:16.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:16.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:16.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:17.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:17.772 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:29:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:29:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:17.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:18.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:19.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:19.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:19.186 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:29:19.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:29:19.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:19.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:19.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:20.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:20.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:20.699 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:29:20.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:29:20.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:20.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:21.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:22.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:22.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:22.558 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:29:22.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:29:22.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:22.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:29:22.892 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/357099826,v1:127.0.0.1:6803/357099826] [v2:127.0.0.1:6804/357099826,v1:127.0.0.1:6805/357099826] exists,up cf70138b-1eed-47b0-a906-48883c6b49ff 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:29:22.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:29:22.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:29:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:29:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:29:22.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:29:22.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:29:22.899 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:29:22.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:29:22.899 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:29:22.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 66a22431-7de0-4af6-a4ed-a1e7b35b92ad' 2024-06-11T17:29:22.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:29:22.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByiWhmJ7y+NhAARfrq/WnRyWe0FK+XZvoErw== 2024-06-11T17:29:22.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByiWhmJ7y+NhAARfrq/WnRyWe0FK+XZvoErw=="}' 2024-06-11T17:29:22.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66a22431-7de0-4af6-a4ed-a1e7b35b92ad -i td/osd-scrub-repair/1/new.json 2024-06-11T17:29:23.362 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:29:23.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:29:23.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQByiWhmJ7y+NhAARfrq/WnRyWe0FK+XZvoErw== --osd-uuid 66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:29:23.427 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.425+0000 7f565f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:23.463 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.461+0000 7f565f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:23.465 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.465+0000 7f565f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:23.465 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.465+0000 7f565f0537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:29:23.465 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.465+0000 7f565f0537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:29:23.466 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.465+0000 7f565f0537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:29:23.535 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:23.533+0000 7f565f0537c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:29:30.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:29:30.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:29:30.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:29:30.172 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:29:30.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:29:30.589 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:29:30.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:29:30.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:29:30.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:29:30.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:29:30.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:29:30.627 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:30.625+0000 7fb3c4c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:30.653 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:30.653+0000 7fb3c4c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:30.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:30.653+0000 7fb3c4c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:29:30.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:30.943 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:29:30.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:29:30.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:30.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:31.206 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:31.205+0000 7fb3c4c907c0 -1 Falling back to public interface 2024-06-11T17:29:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:29:32.304 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:29:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:32.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:32.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:33.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:33.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:33.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:29:33.642 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:29:33.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:33.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:33.860 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:29:33.857+0000 7fb3c4c907c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:29:33.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:34.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:34.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:34.976 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:29:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:29:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:36.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:36.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:36.311 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:29:36.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:29:36.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:36.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:36.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:37.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:37.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:37.683 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:29:37.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:29:37.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:37.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:38.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:39.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:39.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:39.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:29:39.020 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:29:39.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:39.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:40.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:40.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:40.358 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:29:40.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:29:40.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:40.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:40.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:41.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:41.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:41.730 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:29:41.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:29:41.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:41.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:42.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:29:43.110 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:29:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:43.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:44.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:44.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:44.500 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:29:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:29:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:44.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:45.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:45.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:45.933 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:29:45.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:29:45.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:45.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:46.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:29:47.280 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:29:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:47.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:48.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:48.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:48.704 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:29:48.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:29:48.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:48.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:49.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:50.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:50.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:50.488 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:29:50.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:29:50.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:50.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:51.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:52.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:52.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:52.592 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:29:52.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:29:52.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:52.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:53.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:54.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:54.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:54.041 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:29:54.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:29:54.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:54.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:54.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:55.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:55.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:55.943 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:29:55.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:29:55.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:55.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:56.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:57.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:57.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:57.281 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:29:57.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:29:57.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:57.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:58.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:29:59.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:29:59.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:29:59.015 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:29:59.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:29:59.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:29:59.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:29:59.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:00.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:00.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:00.975 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:30:00.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:30:00.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:00.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:30:01.315 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2397729317,v1:127.0.0.1:6811/2397729317] [v2:127.0.0.1:6812/2397729317,v1:127.0.0.1:6813/2397729317] exists,up 66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:30:01.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:30:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:30:01.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:30:01.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:30:01.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c78284c-25ff-4a27-80ff-2a8a906beac9 2024-06-11T17:30:01.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1c78284c-25ff-4a27-80ff-2a8a906beac9' 2024-06-11T17:30:01.322 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 1c78284c-25ff-4a27-80ff-2a8a906beac9 2024-06-11T17:30:01.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:30:01.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZiWhmN9c8FBAAbXWhLwVQ2rMrt1iMhRs82g== 2024-06-11T17:30:01.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZiWhmN9c8FBAAbXWhLwVQ2rMrt1iMhRs82g=="}' 2024-06-11T17:30:01.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c78284c-25ff-4a27-80ff-2a8a906beac9 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:30:01.736 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:30:01.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:30:01.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCZiWhmN9c8FBAAbXWhLwVQ2rMrt1iMhRs82g== --osd-uuid 1c78284c-25ff-4a27-80ff-2a8a906beac9 2024-06-11T17:30:01.791 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.788+0000 7f7883a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:01.856 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.852+0000 7f7883a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:01.858 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.856+0000 7f7883a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:01.858 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.856+0000 7f7883a897c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:30:01.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.856+0000 7f7883a897c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:30:01.859 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.856+0000 7f7883a897c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:30:01.939 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:01.936+0000 7f7883a897c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:30:09.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:30:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:30:09.779 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:30:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:30:09.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:30:10.225 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:30:10.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:30:10.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:30:10.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:30:10.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:30:10.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:30:10.264 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:10.264+0000 7fc7b3e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:10.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:10.296+0000 7fc7b3e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:10.300 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:10.300+0000 7fc7b3e207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:30:10.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:30:10.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:10.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:10.889 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:10.888+0000 7fc7b3e207c0 -1 Falling back to public interface 2024-06-11T17:30:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:12.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:12.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:12.110 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:30:12.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:30:12.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:12.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:12.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:13.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:13.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:13.442 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:30:13.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:30:13.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:13.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:13.517 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:30:13.516+0000 7fc7b3e207c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:30:13.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:14.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:14.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:14.777 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:30:14.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:30:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:15.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:16.117 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:30:16.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:30:16.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:16.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:16.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:17.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:17.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:17.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:30:17.453 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:30:17.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:17.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:17.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:18.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:18.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:18.791 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:30:18.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:30:18.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:18.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:19.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:20.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:20.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:20.130 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:30:20.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:30:20.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:20.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:20.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:21.462 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:30:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:30:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:21.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:22.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:22.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:22.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:30:22.951 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:30:22.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:22.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:23.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:24.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:24.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:24.289 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:30:24.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:30:24.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:24.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:24.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:25.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:25.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:25.707 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:30:25.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:30:25.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:25.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:26.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:27.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:27.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:27.091 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:30:27.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:30:27.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:27.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:27.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:28.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:28.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:28.714 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:30:28.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:30:28.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:28.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:29.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:30.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:30.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:30.386 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:30:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:30:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:31.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:32.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:32.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:32.202 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:30:32.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:30:32.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:32.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:33.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:34.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:34.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:34.181 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:30:34.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:30:34.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:34.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:34.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:35.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:35.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:35.986 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:30:35.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:30:35.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:35.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:36.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:37.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:37.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:37.571 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:30:37.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:30:37.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:37.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:37.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:38.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:38.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:38.988 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:30:38.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:30:38.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:38.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:39.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:30:40.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:30:40.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:30:40.355 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:30:40.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:30:40.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:30:40.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:30:40.736 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/847029062,v1:127.0.0.1:6819/847029062] [v2:127.0.0.1:6820/847029062,v1:127.0.0.1:6821/847029062] exists,up 1c78284c-25ff-4a27-80ff-2a8a906beac9 2024-06-11T17:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-11T17:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:30:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:30:41.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:30:41.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:30:41.548 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:30:41.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:30:42.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:30:45.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-11T17:30:45.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:30:45.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:30:45.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:30:45.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:30:45.922 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:30:45.922 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:30:45.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:30:45.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:45.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:30:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-06-11T17:30:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-06-11T17:30:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-06-11T17:30:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:46.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=73014444053 2024-06-11T17:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 73014444053 2024-06-11T17:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-73014444053' 2024-06-11T17:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:30:46.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855372 2024-06-11T17:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855372 2024-06-11T17:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-73014444053 2-150323855372' 2024-06-11T17:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-06-11T17:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:46.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:30:46.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-06-11T17:30:46.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:46.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-06-11T17:30:46.396 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836502 2024-06-11T17:30:46.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-11T17:30:46.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:30:46.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836502 2024-06-11T17:30:46.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:30:47.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:30:47.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:30:48.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836502 2024-06-11T17:30:48.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:30:49.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T17:30:49.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:30:49.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836502 2024-06-11T17:30:49.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:49.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-73014444053 2024-06-11T17:30:49.389 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:49.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:30:49.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-73014444053 2024-06-11T17:30:49.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:49.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=73014444053 2024-06-11T17:30:49.393 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 73014444053 2024-06-11T17:30:49.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 73014444053' 2024-06-11T17:30:49.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:30:49.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 73014444053 -lt 73014444053 2024-06-11T17:30:49.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:49.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855372 2024-06-11T17:30:49.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:49.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:30:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855372 2024-06-11T17:30:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:49.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855372 2024-06-11T17:30:49.767 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 150323855372 2024-06-11T17:30:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2024-06-11T17:30:49.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:30:50.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855372 -lt 150323855372 2024-06-11T17:30:50.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:30:50.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:30:50.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:30:50.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:30:50.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:30:50.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:30:50.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:30:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:30:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:30:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:30:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:30:50.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:30:50.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:30:50.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:30:50.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:30:51.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-06-11T17:30:51.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:30:51.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-11T17:30:51.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:30:51.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:30:52.952 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:30:52.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:30:53.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-06-11T17:30:53.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-11T17:30:54.592 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 2 allow_ec_overwrites to true 2024-06-11T17:30:54.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:30:54.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:30:54.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:30:54.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:30:54.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:30:54.618 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:30:54.618 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:30:54.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:30:54.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:30:54.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:30:54.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:30:54.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:30:54.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:30:54.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:30:54.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:30:54.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:30:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:30:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:30:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:30:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:30:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:55.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:30:55.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-06-11T17:30:55.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-06-11T17:30:55.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-06-11T17:30:55.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:55.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:30:55.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=73014444056 2024-06-11T17:30:55.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 73014444056 2024-06-11T17:30:55.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-73014444056' 2024-06-11T17:30:55.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:30:55.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:30:55.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855374 2024-06-11T17:30:55.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855374 2024-06-11T17:30:55.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-73014444056 2-150323855374' 2024-06-11T17:30:55.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:55.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-06-11T17:30:55.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:55.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:30:55.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:55.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-06-11T17:30:55.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-06-11T17:30:55.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-06-11T17:30:55.845 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836505 2024-06-11T17:30:55.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:30:56.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-06-11T17:30:56.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:56.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-73014444056 2024-06-11T17:30:56.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:56.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:30:56.191 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-73014444056 2024-06-11T17:30:56.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:56.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=73014444056 2024-06-11T17:30:56.192 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 73014444056 2024-06-11T17:30:56.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 73014444056' 2024-06-11T17:30:56.193 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:30:56.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 73014444056 -lt 73014444056 2024-06-11T17:30:56.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:30:56.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855374 2024-06-11T17:30:56.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:30:56.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:30:56.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855374 2024-06-11T17:30:56.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:30:56.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855374 2024-06-11T17:30:56.536 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 150323855374 2024-06-11T17:30:56.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855374' 2024-06-11T17:30:56.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:30:56.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855374 -lt 150323855374 2024-06-11T17:30:56.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:30:56.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:30:56.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:30:57.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:30:57.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:30:57.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:30:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:30:57.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:30:57.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:30:57.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:30:58.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-11T17:30:58.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:30:58.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-11T17:30:58.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-11T17:30:58.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:30:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-11T17:30:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:30:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:30:58.392 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:30:58.392 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:30:58.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-11T17:30:58.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:30:58.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-11T17:30:59.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:30:59.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:30:59.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:30:59.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:30:59.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:30:59.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:30:59.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:31:00.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:31:00.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:31:01.922 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T17:31:01.922 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-11T17:31:01.922 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:31:02.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:31:02.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:31:02.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:31:02.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:31:02.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:31:02.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:31:02.459 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:31:02.459 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:02.459 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:31:02.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:31:02.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:31:02.461 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:31:02.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:31:02.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:31:02.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:31:02.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:31:02.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:31:02.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:31:02.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:02.496+0000 7f874440d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:02.502 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:02.500+0000 7f874440d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:02.504 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:02.500+0000 7f874440d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:31:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:31:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:31:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:31:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:31:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:31:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:31:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:03.051 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:03.048+0000 7f874440d7c0 -1 Falling back to public interface 2024-06-11T17:31:03.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:31:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:31:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:04.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:05.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:05.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:05.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:31:05.558 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:31:05.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:05.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:05.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:05.992 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:05.988+0000 7f874440d7c0 -1 osd.1 57 log_to_monitors true 2024-06-11T17:31:06.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:06.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:06.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:31:06.889 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:31:06.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:06.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:07.218 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 59 last_clean_interval [17,57) [v2:127.0.0.1:6810/3750122406,v1:127.0.0.1:6811/3750122406] [v2:127.0.0.1:6812/3750122406,v1:127.0.0.1:6813/3750122406] exists,up 66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:31:07.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:31:07.219 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:31:07.220 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:31:07.220 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:31:07.220 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:31:07.220 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:31:07.220 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:31:07.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:07.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:31:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-06-11T17:31:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-06-11T17:31:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-06-11T17:31:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:08.012 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:31:08.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=279172874242 2024-06-11T17:31:08.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 279172874242 2024-06-11T17:31:08.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-279172874242' 2024-06-11T17:31:08.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:08.198 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855378 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855378 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-279172874242 2-150323855378' 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-06-11T17:31:08.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:08.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:31:08.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-06-11T17:31:08.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:08.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-06-11T17:31:08.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-06-11T17:31:08.358 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836509 2024-06-11T17:31:08.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:31:08.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836509 2024-06-11T17:31:08.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:31:09.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:31:09.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:31:10.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-06-11T17:31:10.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:10.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-279172874242 2024-06-11T17:31:10.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:10.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:31:10.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-279172874242 2024-06-11T17:31:10.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:10.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=279172874242 2024-06-11T17:31:10.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-06-11T17:31:10.290 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 279172874242 2024-06-11T17:31:10.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:31:10.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 279172874242 -lt 279172874242 2024-06-11T17:31:10.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:10.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855378 2024-06-11T17:31:10.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:10.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:31:10.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855378 2024-06-11T17:31:10.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:10.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855378 2024-06-11T17:31:10.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855378' 2024-06-11T17:31:10.715 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 150323855378 2024-06-11T17:31:10.716 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:31:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855378 -lt 150323855378 2024-06-11T17:31:11.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:31:11.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:11.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:11.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:31:11.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:31:11.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:31:11.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:31:11.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:31:11.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:31:11.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:31:11.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:31:11.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T17:31:11.809 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:31:11.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:11.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:31:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:31:12.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:31:12.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:31:12.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:31:12.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:31:12.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:31:12.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:12.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:13.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:31:13.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:31:13.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:31:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-11T17:31:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:31:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:31:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:31:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:31:13.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-11T17:31:13.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-11T17:31:13.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:13.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:13.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:13.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:13.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:13.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:13.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:14.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:14.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:15.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:15.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:15.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:15.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:15.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:15.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:15.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:15.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:16.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:16.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:16.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:16.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:16.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:16.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:16.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:16.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:16.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:17.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:17.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:17.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:17.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:17.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:17.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:17.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:18.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:18.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:19.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:19.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:19.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:19.115 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:19.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:19.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:19.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:20.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:20.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:20.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:20.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:20.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:20.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:20.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:20.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:09.450086+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:20.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:31:21.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:31:21.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:31:21.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:31:21.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:31:21.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:31:21.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:31:21.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:31:17.635918+0000 '>' 2024-06-11T17:31:09.450086+0000 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:31:22.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:31:22.128 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:31:22.128 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:31:22.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:31:22.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:31:22.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:31:22.232 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:31:22.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:31:22.565 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:31:22.565 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:31:22.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:31:22.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:22.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:31:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-06-11T17:31:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-06-11T17:31:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-06-11T17:31:22.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:22.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:31:22.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=279172874246 2024-06-11T17:31:22.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 279172874246 2024-06-11T17:31:22.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-279172874246' 2024-06-11T17:31:22.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:22.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:31:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855382 2024-06-11T17:31:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855382 2024-06-11T17:31:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-279172874246 2-150323855382' 2024-06-11T17:31:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:23.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-06-11T17:31:23.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:23.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:31:23.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-06-11T17:31:23.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:23.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-06-11T17:31:23.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-11T17:31:23.052 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836513 2024-06-11T17:31:23.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:31:23.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836513 2024-06-11T17:31:23.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:31:24.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:31:24.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:31:24.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-11T17:31:24.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:24.771 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-279172874246 2024-06-11T17:31:24.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:24.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:31:24.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-279172874246 2024-06-11T17:31:24.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:24.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=279172874246 2024-06-11T17:31:24.775 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 279172874246 2024-06-11T17:31:24.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 279172874246' 2024-06-11T17:31:24.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:31:25.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 279172874246 -lt 279172874246 2024-06-11T17:31:25.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:25.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855382 2024-06-11T17:31:25.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:25.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:31:25.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:25.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855382 2024-06-11T17:31:25.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855382 2024-06-11T17:31:25.112 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 150323855382 2024-06-11T17:31:25.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855382' 2024-06-11T17:31:25.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:31:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855382 -lt 150323855382 2024-06-11T17:31:25.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:31:25.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:25.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:25.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:31:25.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:31:25.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:31:26.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:31:26.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:31:26.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:26.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:26.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:31:26.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:31:26.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:31:26.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-11T17:31:26.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-11T17:31:26.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T17:31:26.666 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-11T17:31:26.666 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:31:26.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:31:26.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:31:26.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:31:27.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-11T17:31:27.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:31:27.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:31:27.379 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:31:27.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:31:27.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:31:27.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:31:27.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:31:27.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:31:27.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:31:27.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:31:27.869 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T17:31:27.869 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-11T17:31:27.870 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:31:27.870 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:31:27.870 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:31:28.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:31:28.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:28.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T17:31:28.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:31:28.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:31:28.161 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:31:28.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-11T17:31:28.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:31:28.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:31:28.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:31:28.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:31:28.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:31:28.200 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:28.200+0000 7fd7579307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:28.203 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:28.200+0000 7fd7579307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:28.205 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:28.204+0000 7fd7579307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:28.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:31:28.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:28.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:28.774 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:28.772+0000 7fd7579307c0 -1 Falling back to public interface 2024-06-11T17:31:29.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:30.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:30.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:30.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:31:30.090 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:31:30.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:30.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:30.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:31:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:31:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:31.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:31.806 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:31.804+0000 7fd7579307c0 -1 osd.1 77 log_to_monitors true 2024-06-11T17:31:32.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:32.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:32.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:31:32.786 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:31:32.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:32.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 85 up_thru 65 down_at 79 last_clean_interval [65,77) [v2:127.0.0.1:6810/2828505639,v1:127.0.0.1:6811/2828505639] [v2:127.0.0.1:6812/2828505639,v1:127.0.0.1:6813/2828505639] exists,up 66a22431-7de0-4af6-a4ed-a1e7b35b92ad 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:31:33.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:31:33.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:31:33.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:31:33.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:31:33.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:31:33.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:31:33.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:31:33.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:31:33.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:31:33.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:31:33.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:31:33.589 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:31:33.589 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T17:31:33.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:31:33.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:33.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:31:33.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-06-11T17:31:33.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-06-11T17:31:33.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-06-11T17:31:33.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:33.854 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:31:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=365072220162 2024-06-11T17:31:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 365072220162 2024-06-11T17:31:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-365072220162' 2024-06-11T17:31:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:31:34.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:31:34.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855385 2024-06-11T17:31:34.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855385 2024-06-11T17:31:34.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-365072220162 2-150323855385' 2024-06-11T17:31:34.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:34.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-06-11T17:31:34.341 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:34.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:31:34.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-06-11T17:31:34.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:34.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-06-11T17:31:34.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-11T17:31:34.344 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836516 2024-06-11T17:31:34.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:31:34.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-06-11T17:31:34.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:34.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-365072220162 2024-06-11T17:31:34.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:34.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:31:34.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-365072220162 2024-06-11T17:31:34.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:34.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=365072220162 2024-06-11T17:31:34.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-06-11T17:31:34.741 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 365072220162 2024-06-11T17:31:34.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:31:35.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 365072220162 2024-06-11T17:31:35.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:31:36.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:31:36.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:31:36.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 365072220162 -lt 365072220162 2024-06-11T17:31:36.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:31:36.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:31:36.600 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855385 2024-06-11T17:31:36.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:31:36.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855385 2024-06-11T17:31:36.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:31:36.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855385 2024-06-11T17:31:36.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855385' 2024-06-11T17:31:36.604 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 150323855385 2024-06-11T17:31:36.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:31:37.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855385 -lt 150323855385 2024-06-11T17:31:37.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:31:37.067 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:37.067 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:37.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:31:37.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:31:37.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:31:37.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:31:37.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:31:37.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:31:37.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:31:37.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:31:37.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:31:37.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:31:37.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:31:37.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:31:38.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:31:38.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:31:38.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:31:38.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:31:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:31:38.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:31:38.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:31:38.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:31:38.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:31:38.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:31:38.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:31:38.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:31:38.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:31:38.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:31:38.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:31:38.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:31:38.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:31:38.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:31:38.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:31:38.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:31:38.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:31:38.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:31:38.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:31:38.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:31:38.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:31:38.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:31:38.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:31:38.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:31:38.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:31:38.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:38.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:38.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:31:38.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:31:38.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:31:38.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:31:38.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:31:38.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:31:38.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:31:38.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:31:38.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:31:38.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:31:38.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:31:38.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:31:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:31:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:31:38.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:31:38.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:31:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:31:38.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:31:38.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:31:38.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:31:38.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:38.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:38.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:31:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:31:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:31:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:31:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:31:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:38.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:31:38.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:31:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:31:38.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:31:38.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:31:38.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:31:39.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:31:39.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:31:39.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:31:39.265 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:31:39.265 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:39.265 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:39.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:39.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:31:39.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:31:39.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:31:39.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:31:39.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:31:39.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:39.488 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:31:39.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:31:39.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:31:39.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:31:39.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:31:39.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:31:39.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:31:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:31:39.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:31:39.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:31:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:31:39.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:31:40.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:40.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:40.595 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:31:40.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:31:40.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-11T17:31:40.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:31:40.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-11T17:31:40.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:31:40.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:31:40.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:31:40.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:31:40.638 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:31:40.638 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:31:40.638 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:31:40.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:40.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:31:40.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:31:40.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:31:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7f5788b-3055-43be-8c0a-b24ab901fe71 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7f5788b-3055-43be-8c0a-b24ab901fe71' 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 b7f5788b-3055-43be-8c0a-b24ab901fe71 2024-06-11T17:31:40.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:31:40.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8iWhm2XBNJxAAn+rIaDAoh4wUqXr7rS2/Kw== 2024-06-11T17:31:40.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8iWhm2XBNJxAAn+rIaDAoh4wUqXr7rS2/Kw=="}' 2024-06-11T17:31:40.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7f5788b-3055-43be-8c0a-b24ab901fe71 -i td/osd-scrub-repair/0/new.json 2024-06-11T17:31:40.920 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:31:40.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:31:40.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8iWhm2XBNJxAAn+rIaDAoh4wUqXr7rS2/Kw== --osd-uuid b7f5788b-3055-43be-8c0a-b24ab901fe71 2024-06-11T17:31:40.964 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.960+0000 7f7bba9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:40.987 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.984+0000 7f7bba9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:40.989 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.988+0000 7f7bba9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:40.989 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.988+0000 7f7bba9437c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:31:40.990 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.988+0000 7f7bba9437c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:31:40.990 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:40.988+0000 7f7bba9437c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:31:41.088 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:41.084+0000 7f7bba9437c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:31:47.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:31:47.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:31:47.729 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:31:47.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:31:47.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:31:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:31:48.147 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:31:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:31:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:31:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:31:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:31:48.174 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:48.172+0000 7fcfdadeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:48.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:48.176+0000 7fcfdadeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:48.178 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:48.176+0000 7fcfdadeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:31:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:31:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:31:48.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:31:48.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:48.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:48.718 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:48.716+0000 7fcfdadeb7c0 -1 Falling back to public interface 2024-06-11T17:31:48.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:49.840 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:31:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:31:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:50.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:51.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:51.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:51.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:31:51.174 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:31:51.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:51.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:51.425 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:31:51.424+0000 7fcfdadeb7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:31:51.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:52.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:52.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:52.554 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:31:52.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:31:52.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:52.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:52.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:53.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:53.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:53.889 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:31:53.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:31:53.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:53.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:55.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:55.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:55.360 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:31:55.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:31:55.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:55.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:55.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:56.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:56.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:56.700 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:31:56.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:31:56.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:56.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:57.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:58.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:58.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:58.108 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:31:58.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:31:58.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:58.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:58.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:31:59.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:31:59.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:31:59.531 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:31:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:31:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:31:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:31:59.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:00.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:00.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:00.871 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:32:00.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:32:00.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:00.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:01.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:02.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:02.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:02.314 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:32:02.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:32:02.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:02.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:02.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:03.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:03.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:03.659 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:32:03.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:32:03.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:03.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:03.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:04.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:04.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:04.998 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:32:04.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:32:04.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:04.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:05.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:06.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:06.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:06.816 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:32:06.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:32:06.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:06.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:07.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:08.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:32:08.498 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:32:08.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:08.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:09.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:10.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:10.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:10.061 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:32:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:32:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:10.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:11.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:11.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:11.702 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:32:11.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:32:11.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:11.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:12.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:13.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:13.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:13.489 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:32:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:32:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:14.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:15.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:15.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:15.420 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:32:15.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:32:15.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:15.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:15.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:16.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:16.764 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:32:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:32:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:32:17.098 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/130575531,v1:127.0.0.1:6803/130575531] [v2:127.0.0.1:6804/130575531,v1:127.0.0.1:6805/130575531] exists,up b7f5788b-3055-43be-8c0a-b24ab901fe71 2024-06-11T17:32:17.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:32:17.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:32:17.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:32:17.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:32:17.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:32:17.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:32:17.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:32:17.105 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:32:17.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b2d8681e-dbb3-48ae-bf03-7e16e7c4930a' 2024-06-11T17:32:17.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:32:17.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhimhmjxN2BxAAhqztQ2vWw0GTbJ0VNRYJ8g== 2024-06-11T17:32:17.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhimhmjxN2BxAAhqztQ2vWw0GTbJ0VNRYJ8g=="}' 2024-06-11T17:32:17.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2d8681e-dbb3-48ae-bf03-7e16e7c4930a -i td/osd-scrub-repair/1/new.json 2024-06-11T17:32:17.513 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:32:17.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:32:17.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhimhmjxN2BxAAhqztQ2vWw0GTbJ0VNRYJ8g== --osd-uuid b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:32:17.575 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.571+0000 7febf9c137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:17.598 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.595+0000 7febf9c137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:17.599 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.595+0000 7febf9c137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:17.600 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.595+0000 7febf9c137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:32:17.600 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.595+0000 7febf9c137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:32:17.600 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.595+0000 7febf9c137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:32:17.672 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:17.667+0000 7febf9c137c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:32:24.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:32:24.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:32:24.047 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:32:24.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:32:24.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:32:24.460 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:32:24.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:32:24.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:32:24.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:32:24.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:32:24.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:32:24.486 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:24.483+0000 7f9913ded7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:24.488 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:24.487+0000 7f9913ded7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:24.490 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:24.487+0000 7f9913ded7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:24.835 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:32:24.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:32:24.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:24.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:25.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:25.047+0000 7f9913ded7c0 -1 Falling back to public interface 2024-06-11T17:32:25.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:26.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:26.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:26.204 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:32:26.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:32:26.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:26.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:26.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:27.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:27.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:27.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:32:27.539 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:32:27.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:27.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:27.570 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:27.567+0000 7f9913ded7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:32:27.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:28.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:28.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:28.966 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:32:28.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:32:28.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:28.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:29.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:30.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:30.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:30.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:32:30.304 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:32:30.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:30.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:30.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:31.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:31.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:31.639 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:32:31.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:32:31.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:31.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:32.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:33.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:33.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:33.076 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:32:33.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:32:33.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:33.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:33.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:34.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:34.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:34.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:32:34.409 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:32:34.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:34.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:34.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:35.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:35.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:35.748 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:32:35.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:32:35.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:35.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:36.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:37.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:37.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:37.093 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:32:37.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:32:37.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:37.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:37.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:38.422 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:32:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:32:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:38.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:39.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:39.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:39.839 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:32:39.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:32:39.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:39.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:40.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:41.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:41.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:41.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:32:41.184 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:32:41.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:41.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:41.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:42.651 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:32:42.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:32:42.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:42.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:43.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:44.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:44.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:44.179 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:32:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:32:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:44.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:45.695 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:32:45.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:32:45.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:45.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:46.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:47.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:47.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:47.413 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:32:47.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:32:47.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:47.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:47.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:48.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:48.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:48.785 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:32:48.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:32:48.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:48.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:49.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:50.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:50.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:50.656 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:32:50.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:32:50.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:50.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:51.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:52.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:52.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:52.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:32:52.269 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:32:52.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:52.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:52.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:32:53.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:32:53.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:32:53.606 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:32:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:32:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:32:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:32:54.003 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3740780991,v1:127.0.0.1:6811/3740780991] [v2:127.0.0.1:6812/3740780991,v1:127.0.0.1:6813/3740780991] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:32:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:32:54.008 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:32:54.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:32:54.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:32:54.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:32:54.010 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:32:54.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:32:54.010 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:32:54.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4fb9607a-0f4b-458d-94d2-683217aa3604' 2024-06-11T17:32:54.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:32:54.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGimhmX+gEAhAAklIli33S+lwHTtPn098E3Q== 2024-06-11T17:32:54.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGimhmX+gEAhAAklIli33S+lwHTtPn098E3Q=="}' 2024-06-11T17:32:54.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4fb9607a-0f4b-458d-94d2-683217aa3604 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:32:54.445 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:32:54.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:32:54.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGimhmX+gEAhAAklIli33S+lwHTtPn098E3Q== --osd-uuid 4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:32:54.505 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.503+0000 7f3b75cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:54.558 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.551+0000 7f3b75cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:54.558 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.555+0000 7f3b75cd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:32:54.558 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.555+0000 7f3b75cd17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:32:54.558 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.555+0000 7f3b75cd17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:32:54.558 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.555+0000 7f3b75cd17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:32:54.652 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:32:54.647+0000 7f3b75cd17c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:33:01.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:33:01.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:33:01.232 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:33:01.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:33:01.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:33:01.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:33:01.668 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:33:01.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:33:01.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:33:01.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:33:01.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:33:01.702 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:01.699+0000 7f649aea87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:01.734 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:01.731+0000 7f649aea87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:01.735 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:01.731+0000 7f649aea87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:02.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:33:02.014 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:33:02.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:02.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:02.287 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:02.283+0000 7f649aea87c0 -1 Falling back to public interface 2024-06-11T17:33:02.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:03.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:03.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:03.370 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:33:03.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:33:03.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:03.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:03.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:04.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:04.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:04.697 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:33:04.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:33:04.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:04.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:04.938 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:04.935+0000 7f649aea87c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:33:05.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:06.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:06.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:06.041 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:33:06.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:33:06.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:06.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:06.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:07.380 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:33:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:33:07.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:07.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:07.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:08.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:08.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:08.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:33:08.903 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:33:08.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:08.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:09.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:10.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:10.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:10.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:33:10.240 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:33:10.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:10.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:10.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:11.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:11.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:11.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:33:11.611 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:33:11.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:11.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:12.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:13.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:13.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:13.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:33:13.093 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:33:13.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:13.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:13.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:14.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:14.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:14.429 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:33:14.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:33:14.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:14.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:14.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:15.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:15.845 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:33:15.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:33:15.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:15.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:16.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:17.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:17.267 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:33:17.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:33:17.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:17.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:17.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:18.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:18.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:18.607 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:33:18.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:33:18.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:18.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:19.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:20.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:20.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:20.003 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:33:20.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:33:20.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:20.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:20.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:21.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:21.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:21.794 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:33:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:33:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:21.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:22.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:23.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:23.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:23.268 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:33:23.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:33:23.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:23.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:23.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:24.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:24.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:24.980 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:33:24.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:33:24.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:24.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:25.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:26.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:26.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:26.696 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:33:26.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:33:26.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:26.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:27.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:28.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:28.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:28.279 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:33:28.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:33:28.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:28.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:28.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:29.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:29.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:29.948 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:33:29.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:33:29.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:29.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:30.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:31.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:31.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:31.484 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:33:31.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:33:31.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:31.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:33:31.835 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2692494400,v1:127.0.0.1:6819/2692494400] [v2:127.0.0.1:6820/2692494400,v1:127.0.0.1:6821/2692494400] exists,up 4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:33:31.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:33:31.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:33:31.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:33:31.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:33:31.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:33:31.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:33:31.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:33:31.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:33:31.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:33:31.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:33:31.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:33:31.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:33:31.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:33:31.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bcef9271-fd5d-48a0-8a58-99f9612def0d' 2024-06-11T17:33:31.843 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:33:31.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:33:31.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrimhmFuedMxAA6sCfXwHDnFyOXHPxE/fXRw== 2024-06-11T17:33:31.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrimhmFuedMxAA6sCfXwHDnFyOXHPxE/fXRw=="}' 2024-06-11T17:33:31.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcef9271-fd5d-48a0-8a58-99f9612def0d -i td/osd-scrub-repair/3/new.json 2024-06-11T17:33:32.262 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:33:32.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-11T17:33:32.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrimhmFuedMxAA6sCfXwHDnFyOXHPxE/fXRw== --osd-uuid bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:33:32.319 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.315+0000 7f2808f477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:32.381 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.379+0000 7f2808f477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:32.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.379+0000 7f2808f477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:32.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.379+0000 7f2808f477c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:33:32.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.379+0000 7f2808f477c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:33:32.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.379+0000 7f2808f477c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:33:32.488 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:32.483+0000 7f2808f477c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-11T17:33:39.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-11T17:33:39.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:33:39.314 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:33:39.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:33:39.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:33:39.762 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:33:39.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:33:39.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:33:39.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:33:39.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:33:39.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:33:39.796 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:39.791+0000 7fd35143f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:39.822 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:39.819+0000 7fd35143f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:39.824 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:39.819+0000 7fd35143f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:33:40.136 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:33:40.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:40.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:40.359 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:40.355+0000 7fd35143f7c0 -1 Falling back to public interface 2024-06-11T17:33:40.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:33:41.500 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:33:41.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:41.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:41.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:42.883 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:33:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:33:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:42.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:43.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:33:43.047+0000 7fd35143f7c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:33:43.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:44.299 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:33:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:33:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:44.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:45.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:45.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:45.638 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:33:45.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:33:45.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:45.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:46.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:47.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:47.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:47.071 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:33:47.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:33:47.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:47.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:47.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:48.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:48.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:48.400 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:33:48.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:33:48.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:48.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:49.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:49.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:49.756 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:33:49.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:33:49.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:49.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:50.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:51.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:51.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:51.091 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:33:51.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:33:51.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:51.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:51.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:52.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:52.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:52.434 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:33:52.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:33:52.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:52.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:52.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:53.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:53.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:53.796 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:33:53.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:33:53.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:53.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:54.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:55.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:55.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:55.125 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:33:55.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:33:55.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:55.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:55.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:33:56.463 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:33:56.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:56.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:56.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:57.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:57.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:57.870 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:33:57.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:33:57.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:57.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:33:58.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:33:59.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:33:59.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:33:59.602 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:33:59.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:33:59.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:33:59.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:00.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:01.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:01.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:01.168 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:34:01.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:34:01.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:01.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:01.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:02.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:02.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:02.769 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:34:02.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:34:02.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:02.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:03.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:04.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:04.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:04.404 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:34:04.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:34:04.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:04.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:04.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:05.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:05.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:05.852 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:34:05.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:34:05.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:05.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:06.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:07.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:07.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:07.415 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:34:07.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:34:07.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:07.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:07.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:08.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:08.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:08.836 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:34:08.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:34:08.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:08.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:09.174 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2365038620,v1:127.0.0.1:6827/2365038620] [v2:127.0.0.1:6828/2365038620,v1:127.0.0.1:6829/2365038620] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:34:09.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:34:09.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:34:09.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:34:09.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-11T17:34:09.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:34:09.547 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:34:09.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:34:09.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:34:10.025 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:34:10.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:34:11.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:34:13.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-11T17:34:13.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:34:13.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:34:13.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:34:13.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:34:13.250 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:34:13.250 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:34:13.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:34:13.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:34:13.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:34:13.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:34:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:34:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:34:13.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:34:13.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:13.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:34:13.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-06-11T17:34:13.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-06-11T17:34:13.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-06-11T17:34:13.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:13.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:34:14.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-06-11T17:34:14.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-06-11T17:34:14.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672979' 2024-06-11T17:34:14.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:14.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:34:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-06-11T17:34:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-06-11T17:34:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672979 2-64424509456' 2024-06-11T17:34:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:14.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:34:14.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-06-11T17:34:14.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-06-11T17:34:14.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672979 2-64424509456 3-85899345932' 2024-06-11T17:34:14.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:14.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:14.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-06-11T17:34:14.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:34:14.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-06-11T17:34:14.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:14.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-06-11T17:34:14.327 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836510 2024-06-11T17:34:14.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-06-11T17:34:14.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:14.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836510 2024-06-11T17:34:14.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:34:15.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:34:15.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:15.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-06-11T17:34:15.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:15.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:15.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-06-11T17:34:15.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:34:15.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-06-11T17:34:15.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:15.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-06-11T17:34:15.974 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672979 2024-06-11T17:34:15.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-11T17:34:15.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:34:16.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672979 2024-06-11T17:34:16.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:16.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-06-11T17:34:16.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:16.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:34:16.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-06-11T17:34:16.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:16.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-06-11T17:34:16.330 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509456 2024-06-11T17:34:16.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-11T17:34:16.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:34:16.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-11T17:34:16.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:16.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:16.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345932 2024-06-11T17:34:16.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:34:16.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345932 2024-06-11T17:34:16.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:16.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-06-11T17:34:16.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-06-11T17:34:16.659 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345932 2024-06-11T17:34:16.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:34:17.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-06-11T17:34:17.002 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:34:17.003 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:17.003 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:17.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:34:17.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:34:17.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:34:17.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:34:17.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:34:17.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:34:17.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:34:17.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:34:17.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:34:17.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:34:17.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:17.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:18.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:34:18.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:34:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-11T17:34:18.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:34:18.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:34:19.497 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:34:19.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:34:20.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:34:20.523 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:34:20.524 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:34:20.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:34:20.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:34:20.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:34:20.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:34:20.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:34:20.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:34:20.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:34:20.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:34:20.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:34:21.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:34:21.102 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:21.102 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:21.102 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:34:21.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:34:21.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:21.103 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:34:21.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-06-11T17:34:21.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-06-11T17:34:21.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-06-11T17:34:21.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:21.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:34:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-06-11T17:34:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-06-11T17:34:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672982' 2024-06-11T17:34:21.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:21.528 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:34:21.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509459 2024-06-11T17:34:21.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509459 2024-06-11T17:34:21.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672982 2-64424509459' 2024-06-11T17:34:21.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:21.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:34:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345934 2024-06-11T17:34:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345934 2024-06-11T17:34:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672982 2-64424509459 3-85899345934' 2024-06-11T17:34:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:21.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-06-11T17:34:21.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:21.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:34:21.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-06-11T17:34:21.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:21.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-06-11T17:34:21.936 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836512 2024-06-11T17:34:21.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-06-11T17:34:21.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:22.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-06-11T17:34:22.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:22.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-06-11T17:34:22.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:34:22.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-06-11T17:34:22.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-06-11T17:34:22.286 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672982 2024-06-11T17:34:22.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-11T17:34:22.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:34:22.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-06-11T17:34:22.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:22.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509459 2024-06-11T17:34:22.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:22.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:34:22.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-06-11T17:34:22.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:22.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509459 2024-06-11T17:34:22.620 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509459 2024-06-11T17:34:22.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-11T17:34:22.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:34:22.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509459 2024-06-11T17:34:22.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:34:23.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:34:23.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:34:24.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-11T17:34:24.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:24.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345934 2024-06-11T17:34:24.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:24.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:34:24.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345934 2024-06-11T17:34:24.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345934 2024-06-11T17:34:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-06-11T17:34:24.299 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345934 2024-06-11T17:34:24.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:34:24.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345934 -lt 85899345934 2024-06-11T17:34:24.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:34:24.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:24.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:25.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:34:25.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:34:25.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:34:25.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:34:25.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:34:25.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:25.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:25.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:34:25.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:34:25.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:34:25.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T17:34:25.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T17:34:26.266 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:34:26.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T17:34:26.718 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:34:26.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T17:34:26.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T17:34:26.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:34:27.051 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-11T17:34:27.051 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:34:27.051 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:34:27.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:34:27.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:34:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-11T17:34:27.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-11T17:34:27.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T17:34:27.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-11T17:34:27.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-11T17:34:27.386 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:34:27.386 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T17:34:27.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-11T17:34:27.732 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:27.732 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:27.732 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T17:34:27.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-11T17:34:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:34:27.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:34:27.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:34:27.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:34:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:34:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:34:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:34:28.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:34:28.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:34:28.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:34:30.129 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:34:30.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:34:30.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:34:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:34:30.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:34:30.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:34:30.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:34:30.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:34:30.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:34:30.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:34:30.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:34:30.667 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:34:30.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:34:30.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:34:30.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:34:30.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:34:30.669 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:34:30.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:34:30.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:34:30.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:34:30.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:34:30.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:34:30.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:34:30.703 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:30.702+0000 7f9db89567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:30.705 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:30.706+0000 7f9db89567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:30.707 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:30.706+0000 7f9db89567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:31.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:31.239 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:31.238+0000 7f9db89567c0 -1 Falling back to public interface 2024-06-11T17:34:31.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:32.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:32.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:32.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:34:32.361 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:32.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:32.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:32.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:33.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:33.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:33.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:34:33.700 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:33.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:33.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:34.046 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:34.046+0000 7f9db89567c0 -1 osd.3 35 log_to_monitors true 2024-06-11T17:34:34.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:35.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:35.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:35.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:34:35.058 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:34:35.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:35.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:35.448 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/1499948124,v1:127.0.0.1:6827/1499948124] [v2:127.0.0.1:6828/1499948124,v1:127.0.0.1:6829/1499948124] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:34:35.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:34:35.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:34:35.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:34:35.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:34:35.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:34:35.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:34:35.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:34:35.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:34:35.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:34:35.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:34:35.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:34:35.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:34:35.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:34:35.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:34:36.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:36.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:34:36.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-06-11T17:34:36.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-06-11T17:34:36.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-06-11T17:34:36.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:36.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:34:36.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672986 2024-06-11T17:34:36.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672986 2024-06-11T17:34:36.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672986' 2024-06-11T17:34:36.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:36.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:34:36.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509463 2024-06-11T17:34:36.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509463 2024-06-11T17:34:36.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509463' 2024-06-11T17:34:36.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:36.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:34:36.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=167503724546 2024-06-11T17:34:36.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 167503724546 2024-06-11T17:34:36.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509463 3-167503724546' 2024-06-11T17:34:36.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:36.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-06-11T17:34:36.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:36.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:34:36.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-06-11T17:34:36.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:36.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-06-11T17:34:36.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-11T17:34:36.768 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836516 2024-06-11T17:34:36.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:37.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836516 2024-06-11T17:34:37.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:34:38.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:34:38.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:38.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836517 -lt 21474836516 2024-06-11T17:34:38.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:38.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672986 2024-06-11T17:34:38.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:38.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:34:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-06-11T17:34:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:38.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672986 2024-06-11T17:34:38.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-11T17:34:38.429 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672986 2024-06-11T17:34:38.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:34:38.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672986 2024-06-11T17:34:38.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:38.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509463 2024-06-11T17:34:38.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:38.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:34:38.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509463 2024-06-11T17:34:38.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509463 2024-06-11T17:34:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-06-11T17:34:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509463 2024-06-11T17:34:38.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:34:39.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509463 -lt 64424509463 2024-06-11T17:34:39.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:39.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-167503724546 2024-06-11T17:34:39.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:39.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:34:39.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-167503724546 2024-06-11T17:34:39.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:39.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=167503724546 2024-06-11T17:34:39.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 167503724546' 2024-06-11T17:34:39.126 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 167503724546 2024-06-11T17:34:39.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:34:39.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724546 -lt 167503724546 2024-06-11T17:34:39.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:34:39.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:39.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:39.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:34:39.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:34:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:34:40.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:34:40.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:34:40.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:40.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:40.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:34:41.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:34:41.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:34:41.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:34:41.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:34:41.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:34:41.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:34:41.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:34:41.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:41.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:34:41.674 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:34:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:34:41.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:34:41.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:34:41.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:34:41.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:34:41.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:34:42.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:19.338615+0000 '>' 2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:42.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:34:43.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:34:43.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:34:43.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:34:43.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:34:43.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:34:43.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:34:43.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:34:43.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:19.338615+0000 '>' 2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:43.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:34:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:34:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:34:44.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:34:44.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:34:44.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:34:44.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:34:44.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:34:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:19.338615+0000 '>' 2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:34:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:34:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:34:45.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:34:45.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:34:45.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:34:45.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:34:45.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:34:46.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:19.338615+0000 2024-06-11T17:34:46.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:34:46.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:34:46.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:34:46.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:34:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:34:46.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:34:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:34:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:34:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:34:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:34:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:34:46.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:34:46.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:34:46.763 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:34:46.763 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:34:46.763 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:34:46.764 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:34:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:34:47.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:34:47.053 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:34:47.053 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:34:47.053 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:34:47.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:34:47.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:34:47.055 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:34:47.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:34:47.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:34:47.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:34:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:34:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:34:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:34:47.090 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:47.090+0000 7fbafdf067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:47.093 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:47.094+0000 7fbafdf067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:47.094 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:47.094+0000 7fbafdf067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:47.635 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:47.634+0000 7fbafdf067c0 -1 Falling back to public interface 2024-06-11T17:34:48.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:34:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:49.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:49.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:49.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:50.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:50.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:50.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:34:50.386 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:50.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:50.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:50.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:34:50.766 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:34:50.766+0000 7fbafdf067c0 -1 osd.3 41 log_to_monitors true 2024-06-11T17:34:51.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:34:51.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:34:51.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:34:51.721 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:34:51.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:34:51.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [39,41) [v2:127.0.0.1:6826/3113981652,v1:127.0.0.1:6827/3113981652] [v2:127.0.0.1:6828/3113981652,v1:127.0.0.1:6829/3113981652] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:34:52.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:34:52.209 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:34:52.209 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:34:52.209 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:34:52.209 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:34:52.209 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:34:52.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:34:52.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:34:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:34:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:34:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:34:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:34:52.315 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:34:52.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:52.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:34:52.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836521 2024-06-11T17:34:52.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836521 2024-06-11T17:34:52.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521' 2024-06-11T17:34:52.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:52.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:34:53.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672990 2024-06-11T17:34:53.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672990 2024-06-11T17:34:53.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672990' 2024-06-11T17:34:53.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:53.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:34:53.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509467 2024-06-11T17:34:53.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509467 2024-06-11T17:34:53.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672990 2-64424509467' 2024-06-11T17:34:53.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:34:53.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495618 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495618 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672990 2-64424509467 3-197568495618' 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836521 2024-06-11T17:34:53.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:53.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:34:53.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836521 2024-06-11T17:34:53.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:53.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836521 2024-06-11T17:34:53.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-06-11T17:34:53.309 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836521 2024-06-11T17:34:53.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:53.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836519 -lt 21474836521 2024-06-11T17:34:53.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:34:54.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:34:54.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:34:54.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836521 -lt 21474836521 2024-06-11T17:34:54.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:54.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672990 2024-06-11T17:34:54.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:54.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:34:54.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672990 2024-06-11T17:34:54.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:54.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672990 2024-06-11T17:34:54.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-06-11T17:34:54.951 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672990 2024-06-11T17:34:54.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:34:55.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672990 -lt 42949672990 2024-06-11T17:34:55.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:55.315 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509467 2024-06-11T17:34:55.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:34:55.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509467 2024-06-11T17:34:55.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:55.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509467 2024-06-11T17:34:55.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-06-11T17:34:55.320 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509467 2024-06-11T17:34:55.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:34:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509467 -lt 64424509467 2024-06-11T17:34:55.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:34:55.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-197568495618 2024-06-11T17:34:55.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:34:55.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:34:55.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-197568495618 2024-06-11T17:34:55.705 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:34:55.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495618 2024-06-11T17:34:55.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-06-11T17:34:55.706 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 197568495618 2024-06-11T17:34:55.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:34:56.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495618 -lt 197568495618 2024-06-11T17:34:56.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:34:56.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:56.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:34:56.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:34:56.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:34:56.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:34:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:34:56.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:34:56.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:56.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:34:57.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:34:57.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:34:57.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:34:57.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:34:57.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:34:57.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:34:57.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:34:57.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:34:57.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:34:57.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:34:57.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:34:57.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:34:57.651 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:34:58.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:34:58.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:34:58.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:34:58.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:34:58.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:34:58.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:34:58.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:34:58.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:34:58.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:35:00.698 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:01.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:01.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:01.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:35:01.243 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:35:01.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:35:01.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:35:01.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:35:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:01.278 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:01.278+0000 7f034d4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:01.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:01.282+0000 7f034d4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:01.282 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:01.282+0000 7f034d4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:35:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:35:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:01.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:01.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:01.591 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:35:01.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:01.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:01.836 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:01.834+0000 7f034d4c77c0 -1 Falling back to public interface 2024-06-11T17:35:01.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:02.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:03.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:04.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:04.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:04.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:04.315 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:35:04.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:04.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:04.451 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:04.450+0000 7f034d4c77c0 -1 osd.1 48 log_to_monitors true 2024-06-11T17:35:04.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:06.201 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/820738421,v1:127.0.0.1:6811/820738421] [v2:127.0.0.1:6812/820738421,v1:127.0.0.1:6813/820738421] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:06.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:06.203 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:06.203 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:06.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:06.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:06.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:06.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:35:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:35:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:35:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:06.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836524 2024-06-11T17:35:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836524 2024-06-11T17:35:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524' 2024-06-11T17:35:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:06.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:35:07.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=227633266690 2024-06-11T17:35:07.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 227633266690 2024-06-11T17:35:07.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-227633266690' 2024-06-11T17:35:07.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:07.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:35:07.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-06-11T17:35:07.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-06-11T17:35:07.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-227633266690 2-64424509471' 2024-06-11T17:35:07.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:07.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495622 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495622 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-227633266690 2-64424509471 3-197568495622' 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836524 2024-06-11T17:35:07.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:07.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:35:07.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836524 2024-06-11T17:35:07.416 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836524 2024-06-11T17:35:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-06-11T17:35:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836524 2024-06-11T17:35:07.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:35:07.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836524 2024-06-11T17:35:07.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:07.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-227633266690 2024-06-11T17:35:07.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:07.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:35:07.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-227633266690 2024-06-11T17:35:07.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:07.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=227633266690 2024-06-11T17:35:07.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-06-11T17:35:07.749 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 227633266690 2024-06-11T17:35:07.749 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:35:08.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 227633266690 -lt 227633266690 2024-06-11T17:35:08.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:08.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-06-11T17:35:08.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:08.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:35:08.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-06-11T17:35:08.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:08.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-06-11T17:35:08.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-06-11T17:35:08.078 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509471 2024-06-11T17:35:08.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:35:08.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-06-11T17:35:08.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:08.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-197568495622 2024-06-11T17:35:08.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:08.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:35:08.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-197568495622 2024-06-11T17:35:08.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:08.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495622 2024-06-11T17:35:08.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-06-11T17:35:08.440 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 197568495622 2024-06-11T17:35:08.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:08.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495622 -lt 197568495622 2024-06-11T17:35:08.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:35:08.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:08.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:09.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:35:09.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:09.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:09.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:09.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:09.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:35:09.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:09.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:09.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:35:09.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:09.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:09.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:35:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:35:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:35:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:35:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:35:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:35:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:35:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:10.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:10.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:10.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:35:10.939 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:35:10.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:10.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:10.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:10.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:10.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:10.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:11.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:11.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:12.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:12.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:12.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:12.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:12.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:12.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:12.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:12.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:12.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:13.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:13.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:13.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:13.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:13.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:13.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:13.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:13.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:13.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:14.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:14.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:14.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:14.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:14.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:14.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:14.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:15.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:15.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:16.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:16.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:16.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:16.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:16.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:16.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:16.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:16.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:34:42.116481+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:16.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:17.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:17.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:17.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:17.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:17.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:17.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:17.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:18.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:11.843918+0000 '>' 2024-06-11T17:34:42.116481+0000 2024-06-11T17:35:18.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:35:18.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:35:18.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:35:18.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:35:18.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:35:18.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:35:18.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:35:18.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:35:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:35:18.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:35:18.564 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:35:18.565 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:35:18.565 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:35:18.565 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:35:18.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:35:18.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:18.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:18.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:18.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:18.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:35:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:35:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:35:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:35:18.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:35:18.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:35:18.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:18.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:18.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:18.901 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:18.902+0000 7f1e011e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:19.056 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:19.054+0000 7f1e011e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:19.058 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:19.058+0000 7f1e011e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:19.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:35:19.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:19.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:35:19.603 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:19.602+0000 7f1e011e27c0 -1 Falling back to public interface 2024-06-11T17:35:19.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:20.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:20.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:20.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:20.747 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:35:20.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:20.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:35:21.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:22.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:22.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:22.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:22.080 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:35:22.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:22.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:35:22.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:22.771 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:35:22.770+0000 7f1e011e27c0 -1 osd.3 55 log_to_monitors true 2024-06-11T17:35:23.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:23.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:35:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:35:23.794 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/2490901421,v1:127.0.0.1:6827/2490901421] [v2:127.0.0.1:6828/2490901421,v1:127.0.0.1:6829/2490901421] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:35:23.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:23.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:23.796 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:23.796 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:23.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:23.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:23.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:23.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:23.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:23.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:23.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:23.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:23.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:24.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:24.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836529 2024-06-11T17:35:24.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836529 2024-06-11T17:35:24.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836529' 2024-06-11T17:35:24.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:24.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:35:24.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=227633266694 2024-06-11T17:35:24.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 227633266694 2024-06-11T17:35:24.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836529 1-227633266694' 2024-06-11T17:35:24.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:24.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:35:24.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509475 2024-06-11T17:35:24.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509475 2024-06-11T17:35:24.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836529 1-227633266694 2-64424509475' 2024-06-11T17:35:24.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:24.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:35:24.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=253403070466 2024-06-11T17:35:24.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 253403070466 2024-06-11T17:35:24.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836529 1-227633266694 2-64424509475 3-253403070466' 2024-06-11T17:35:24.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:24.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836529 2024-06-11T17:35:24.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:24.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:35:24.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836529 2024-06-11T17:35:24.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:24.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836529 2024-06-11T17:35:24.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-06-11T17:35:24.865 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836529 2024-06-11T17:35:24.865 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:35:25.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836528 -lt 21474836529 2024-06-11T17:35:25.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:35:26.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:35:26.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:35:26.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836529 -lt 21474836529 2024-06-11T17:35:26.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:26.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:26.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-227633266694 2024-06-11T17:35:26.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:35:26.528 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-227633266694 2024-06-11T17:35:26.528 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:26.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=227633266694 2024-06-11T17:35:26.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-06-11T17:35:26.529 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 227633266694 2024-06-11T17:35:26.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:35:26.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 227633266695 -lt 227633266694 2024-06-11T17:35:26.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:26.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509475 2024-06-11T17:35:26.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:26.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:35:26.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509475 2024-06-11T17:35:26.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:26.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509475 2024-06-11T17:35:26.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-06-11T17:35:26.889 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509475 2024-06-11T17:35:26.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:35:27.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509475 -lt 64424509475 2024-06-11T17:35:27.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:27.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:27.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-253403070466 2024-06-11T17:35:27.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:35:27.232 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-253403070466 2024-06-11T17:35:27.232 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:27.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=253403070466 2024-06-11T17:35:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-06-11T17:35:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 253403070466 2024-06-11T17:35:27.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:27.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070466 -lt 253403070466 2024-06-11T17:35:27.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:35:27.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:27.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:28.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:35:28.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:28.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:28.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:28.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:28.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:35:28.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:28.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:28.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:35:28.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:28.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:28.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:28.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:35:28.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:35:28.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:35:28.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:35:28.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:35:28.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-11T17:35:28.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 156910"' 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 156910' 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/156912: /' 2024-06-11T17:35:28.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 156913"' 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 156913' 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 156910 156913' 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 156910 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:35:28.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/156915: /' 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:35:32.147 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:35:32.148 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:35:32.149 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:32.150 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:32.153 INFO:tasks.workunit.client.0.smithi012.stderr:156912: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: start osd.1 2024-06-11T17:35:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:35:32.272 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.272 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:35:32.272 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T17:35:32.272 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:35:32.272 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:35:32.273 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:32.274 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:32.275 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T17:35:32.275 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:32.275 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:35:32.275 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:32.275 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:32.276 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: start osd.2 2024-06-11T17:35:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:35:36.929 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:35:36.929 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:35:36.929 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:36.929 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:36.929 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2024-06-11T17:35:32.186+0000 7fdbae7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2024-06-11T17:35:32.190+0000 7fdbae7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2024-06-11T17:35:32.190+0000 7fdbae7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:36.930 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 0 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2024-06-11T17:35:32.754+0000 7fdbae7aa7c0 -1 Falling back to public interface 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 1 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:36.931 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2024-06-11T17:35:35.786+0000 7fdbae7aa7c0 -1 osd.1 60 log_to_monitors true 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 3 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:36.932 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:37.145 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2024-06-11T17:35:32.302+0000 7f4f674a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2024-06-11T17:35:32.306+0000 7f4f674a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2024-06-11T17:35:32.306+0000 7f4f674a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:37.146 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 0 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2024-06-11T17:35:32.862+0000 7f4f674a67c0 -1 Falling back to public interface 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 1 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:37.147 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2024-06-11T17:35:36.022+0000 7f4f674a67c0 -1 osd.2 60 log_to_monitors true 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 3 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:37.148 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 4 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: osd.1 up in weight 1 up_from 66 up_thru 66 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/105513345,v1:127.0.0.1:6811/105513345] [v2:127.0.0.1:6812/105513345,v1:127.0.0.1:6813/105513345] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:35:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:39.222 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 1 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 2 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: 3' 2024-06-11T17:35:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836533 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836533 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533' 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperss.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 4 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/3674885814,v1:127.0.0.1:6819/3674885814] [v2:127.0.0.1:6820/3674885814,v1:127.0.0.1:6821/3674885814] exists,up 4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 1 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 2 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: 3' 2024-06-11T17:35:39.226 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836534 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836534 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534' 2024-06-11T17:35:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=283467841539 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 283467841539 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-283467841539' 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808835 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808835 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-283467841539 2-287762808835' 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:35:40.344 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=253403070471 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 253403070471 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-283467841539 2-287762808835 3-253403070471' 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836534 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836534 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836534 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: waiting osd.0 seq 21474836534 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836534 2024-06-11T17:35:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-283467841539 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-283467841539 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=283467841539 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: waiting osd.1 seq 283467841539 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=283467841538 2024-06-11T17:35:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 283467841538 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-283467841538' 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808834 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808834 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-283467841538 2-287762808834' 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=253403070470 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 253403070470 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-283467841538 2-287762808834 3-253403070470' 2024-06-11T17:35:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836533 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836533 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836533 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: waiting osd.0 seq 21474836533 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836533 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-283467841538 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-283467841538 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=283467841538 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: waiting osd.1 seq 283467841538 2024-06-11T17:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 283467841539 -lt 283467841538 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-287762808834 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-287762808834 2024-06-11T17:35:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808834 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: waiting osd.2 seq 287762808834 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808835 -lt 287762808834 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-253403070470 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-253403070470 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=253403070470 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-06-11T17:35:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:156912: waiting osd.3 seq 253403070470 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070469 -lt 253403070470 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070471 -lt 253403070470 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:42.773 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("st0: flush_pg_stats: test 283467841539 -lt 283467841539 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-287762808835 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-287762808835 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808835 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: waiting osd.2 seq 287762808835 2024-06-11T17:35:42.784 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808835 -lt 287762808835 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-253403070471 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-253403070471 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=253403070471 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: waiting osd.3 seq 253403070471 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070469 -lt 253403070471 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:35:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070471 -lt 253403070471 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(containsale") | not)' 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:156912: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:35:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 156913 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:("stale") | not)' 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T17:35:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:35:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:156915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:156915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:35:44.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:35:44.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:35:44.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:35:44.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:35:44.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:35:44.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:35:44.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:35:44.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:44.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:44.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:44.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:45.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:45.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:35:45.354 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:35:45.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:45.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:45.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:45.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:45.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:45.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:45.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:11.843918+0000 '>' 2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:45.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:46.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:46.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:46.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:47.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:11.843918+0000 '>' 2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:47.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:48.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:48.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:48.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:48.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:48.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:48.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:48.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:11.843918+0000 '>' 2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:35:49.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:35:49.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:35:49.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:35:49.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:35:49.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:35:49.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:35:49.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:45.534568+0000 '>' 2024-06-11T17:35:11.843918+0000 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 159883"' 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 159883' 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:35:49.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 159884"' 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 159884' 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 159883 159884' 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 159883 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/159886: /' 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:35:49.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:35:49.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/159888: /' 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:35:50.716 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:35:50.717 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: _ 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: hinfo_key 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: snapset 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:50.718 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:50.721 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:50.722 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:35:50.723 INFO:tasks.workunit.client.0.smithi012.stderr:159886: start osd.1 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T17:35:51.542 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-11T17:35:51.543 INFO:tasks.workunit.client.0.smithi012.stderr:159888: _ 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: hinfo_key 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: snapset 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:35:51.544 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:35:51.546 INFO:tasks.workunit.client.0.smithi012.stderr:159888: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:35:51.547 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T17:35:51.548 INFO:tasks.workunit.client.0.smithi012.stderr:159888: start osd.2 2024-06-11T17:35:55.500 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2024-06-11T17:35:50.758+0000 7fbed190b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2024-06-11T17:35:50.758+0000 7fbed190b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2024-06-11T17:35:50.762+0000 7fbed190b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:55.501 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 0 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2024-06-11T17:35:51.318+0000 7fbed190b7c0 -1 Falling back to public interface 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:55.502 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 1 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:55.503 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2024-06-11T17:35:55.054+0000 7fbed190b7c0 -1 osd.1 70 log_to_monitors true 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 3 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:55.504 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2024-06-11T17:35:51.570+0000 7ff6603837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2024-06-11T17:35:51.574+0000 7ff6603837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2024-06-11T17:35:51.578+0000 7ff6603837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T17:35:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 0 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2024-06-11T17:35:52.106+0000 7ff6603837c0 -1 Falling back to public interface 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:35:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 1 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:35:56.496 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 3 2024-06-11T17:35:56.497 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:56.497 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:56.497 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2024-06-11T17:35:56.066+0000 7ff6603837c0 -1 osd.2 70 log_to_monitors true 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 4 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:35:57.905 INFO:tasks.workunit.client.0.smithi012.stderr:159886: osd.1 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [66,70) [v2:127.0.0.1:6810/164277474,v1:127.0.0.1:6811/164277474] [v2:127.0.0.1:6812/164277474,v1:127.0.0.1:6813/164277474] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:57.906 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 1 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 2 2024-06-11T17:35:57.907 INFO:tasks.workunit.client.0.smithi012.stderr:159886: 3' 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836539 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836539 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539' 2024-06-11T17:35:57.908 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceeph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 4 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:35:58.813 INFO:tasks.workunit.client.0.smithi012.stderr:159888: osd.2 up in weight 1 up_from 78 up_thru 49 down_at 71 last_clean_interval [67,70) [v2:127.0.0.1:6818/3784687359,v1:127.0.0.1:6819/3784687359] [v2:127.0.0.1:6820/3784687359,v1:127.0.0.1:6821/3784687359] exists,up 4fb9607a-0f4b-458d-94d2-683217aa3604 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:35:58.814 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 1 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 2 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: 3' 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:35:58.815 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:35:58.816 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:35:58.816 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836540 2024-06-11T17:35:58.816 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836540 2024-06-11T17:35:58.816 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540' 2024-06-11T17:35:58.816 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:00.205 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalonph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:00.205 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=322122547202 2024-06-11T17:36:00.205 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 322122547202 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-322122547202' 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449090 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449090 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-322122547202 2-335007449090' 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=253403070476 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 253403070476 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-322122547202 2-335007449090 3-253403070476' 2024-06-11T17:36:00.206 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: waiting osd.0 seq 21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836540 -lt 21474836539 2024-06-11T17:36:00.207 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:00.208 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-322122547202 2024-06-11T17:36:00.208 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:00.208 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:00.208 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-322122547202 2024-06-11T17:36:00.208 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:01.080 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /e/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:01.080 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=322122547203 2024-06-11T17:36:01.080 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 322122547203 2024-06-11T17:36:01.080 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-322122547203' 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449091 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449091 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-322122547203 2-335007449091' 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=253403070477 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 253403070477 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-322122547203 2-335007449091 3-253403070477' 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836540 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836540 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836540 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: waiting osd.0 seq 21474836540 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836540 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836540 -lt 21474836540 2024-06-11T17:36:01.082 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:01.083 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-322122547203 2024-06-11T17:36:01.083 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:01.083 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:01.083 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-322122547203 2024-06-11T17:36:01.083 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:01.661 INFO:tasks.workunit.client.0.smithi012.stderr:15988home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=322122547202 2024-06-11T17:36:01.661 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-06-11T17:36:01.661 INFO:tasks.workunit.client.0.smithi012.stderr:159886: waiting osd.1 seq 322122547202 2024-06-11T17:36:01.661 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 322122547203 -lt 322122547202 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449090 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449090 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449090 2024-06-11T17:36:01.662 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: waiting osd.2 seq 335007449090 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449091 -lt 335007449090 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-253403070476 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-253403070476 2024-06-11T17:36:01.663 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=253403070476 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: waiting osd.3 seq 253403070476 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070477 -lt 253403070476 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:01.664 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:01.665 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:01.665 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:01.665 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:02.349 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:36:02.349 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:159886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:36:02.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 159884 2024-06-11T17:36:02.443 INFO:tasks.workunit.client.0.smithi012.stderr:8: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=322122547203 2024-06-11T17:36:02.443 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-06-11T17:36:02.443 INFO:tasks.workunit.client.0.smithi012.stderr:159888: waiting osd.1 seq 322122547203 2024-06-11T17:36:02.443 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 322122547203 -lt 322122547203 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449091 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449091 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449091 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449091' 2024-06-11T17:36:02.444 INFO:tasks.workunit.client.0.smithi012.stderr:159888: waiting osd.2 seq 335007449091 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449091 -lt 335007449091 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-253403070477 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-253403070477 2024-06-11T17:36:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=253403070477 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: waiting osd.3 seq 253403070477 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 253403070477 -lt 253403070477 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:03.152 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:36:03.152 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:03.152 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:159888: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:36:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:36:03.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:36:03.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:36:03.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-11T17:36:03.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:36:03.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:36:03.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 162247"' 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 162247' 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 162250"' 2024-06-11T17:36:03.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 162250' 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 162247 162250' 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 162247 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/162249: /' 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:36:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/162252: /' 2024-06-11T17:36:06.547 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:06.548 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.549 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:36:06.550 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:36:06.550 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:36:06.550 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:36:06.550 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:36:06.550 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:36:06.553 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:36:06.553 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:36:06.554 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:36:06.554 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:36:06.554 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:36:06.554 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:36:06.554 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:36:06.555 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: start osd.3 2024-06-11T17:36:06.556 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:36:06.846 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.846 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:36:06.847 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:06.848 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:36:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:36:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:36:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:36:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:36:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:36:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:162252: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:36:06.851 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:36:06.852 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:36:06.852 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:36:06.852 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:36:06.852 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:36:06.853 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:36:06.853 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:36:06.853 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:36:06.853 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:36:06.853 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:162252: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:06.855 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:06.855 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:06.855 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:36:06.855 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:36:06.855 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:36:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:36:06.857 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:36:06.857 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:36:06.857 INFO:tasks.workunit.client.0.smithi012.stderr:162252: start osd.1 2024-06-11T17:36:06.857 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:36:11.354 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:36:11.354 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2024-06-11T17:36:06.585+0000 7f49583027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2024-06-11T17:36:06.589+0000 7f49583027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2024-06-11T17:36:06.593+0000 7f49583027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 0 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2024-06-11T17:36:07.137+0000 7f49583027c0 -1 Falling back to public interface 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 1 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2024-06-11T17:36:10.093+0000 7f49583027c0 -1 osd.3 79 log_to_monitors true 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 3 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162249: osd.3 up in weight 1 up_from 84 up_thru 78 down_at 80 last_162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2024-06-11T17:36:06.881+0000 7fe5862d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2024-06-11T17:36:06.885+0000 7fe5862d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2024-06-11T17:36:06.885+0000 7fe5862d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:36:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 0 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2024-06-11T17:36:07.429+0000 7fe5862d17c0 -1 Falling back to public interface 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 1 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2024-06-11T17:36:10.733+0000 7fe5862d17c0 -1 osd.1 79 log_to_monitors true 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:11.900 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:36:11.901 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 3 2024-06-11T17:36:11.901 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:11.901 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [59,79) [v2:127.0.0.1:6810/378213494,v1:127.0.0.1:6811/378213494] [v2:127.0.0.1:6812/378213494,v1:127.0.0.1:6813/378213494] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:36:12.777 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:36:12.778 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 1 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 2 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: 3' 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836544 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836544 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544' 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=2 2024-06-11T17:36:12.779 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 2 2024-06-11T17:36:12.780 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-2' 2024-06-11T17:36:12.780 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:12.780 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /hos.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 4 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:14.039 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: osd.1 up in weight 1 up_from 87 up_thru 87 down_at 80 last_clean_interval [75,79) [v2:127.0.0.1:6826/2329988734,v1:127.0.0.1:6827/2329988734] [v2:127.0.0.1:6828/2329988734,v1:127.0.0.1:6829/2329988734] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:36:14.040 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 1 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 2 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: 3' 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:36:14.041 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:14.042 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:36:14.042 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836545 2024-06-11T17:36:14.042 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836545 2024-06-11T17:36:14.042 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545' 2024-06-11T17:36:14.042 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helme/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449095 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449095 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-2 2-335007449095' 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:15.047 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252866 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252866 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-2 2-335007449095 3-360777252866' 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836544 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836544 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836544 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: waiting osd.0 seq 21474836544 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836542 -lt 21474836544 2024-06-11T17:36:15.048 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836544 -lt 21474836544 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: waiting osd.1 seq 2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 2 -lt 2 2024-06-11T17:36:15.049 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:15.050 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449095 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntupers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=373662154755 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 373662154755 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-373662154755' 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449096 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449096 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-373662154755 2-335007449096' 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:16.171 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252867 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252867 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-373662154755 2-335007449096 3-360777252867' 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836545 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836545 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836545 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-06-11T17:36:16.172 INFO:tasks.workunit.client.0.smithi012.stderr:162252: waiting osd.0 seq 21474836545 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836544 -lt 21474836545 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836545 -lt 21474836545 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-373662154755 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-373662154755 2024-06-11T17:36:16.173 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:16.461 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/u/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:16.461 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449095 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449095 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449095' 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: waiting osd.2 seq 335007449095 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449095 -lt 335007449095 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-360777252866 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:16.462 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-360777252866 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252866 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 360777252866' 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: waiting osd.3 seq 360777252866 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252866 -lt 360777252866 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:16.463 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:16.464 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:162249: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:162249: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:36:16.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 162250 2024-06-11T17:36:17.605 INFO:tasks.workunit.client.0.smithi012.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=373662154755 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 373662154755' 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: waiting osd.1 seq 373662154755 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 373662154755 -lt 373662154755 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449096 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449096 2024-06-11T17:36:17.606 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449096 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449096' 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: waiting osd.2 seq 335007449096 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449096 -lt 335007449096 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-360777252867 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-360777252867 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252867 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-06-11T17:36:17.607 INFO:tasks.workunit.client.0.smithi012.stderr:162252: waiting osd.3 seq 360777252867 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252867 -lt 360777252867 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:17.608 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:162252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:18.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:18.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:36:18.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:36:18.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:36:18.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:36:18.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:36:18.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:36:18.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:36:18.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:36:18.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:36:18.362 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:36:18.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:36:18.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:36:18.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:36:19.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:19.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:36:19.357 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:36:19.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:36:19.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:36:19.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:36:19.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:36:19.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:36:19.376 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:36:19.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:45.534568+0000 '>' 2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:19.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:36:20.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:36:20.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:36:20.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:36:20.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:36:20.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:36:20.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:36:20.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:36:21.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:35:45.534568+0000 '>' 2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:21.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:36:22.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:36:22.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:36:22.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:36:22.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:36:22.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:36:22.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:36:22.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:36:20.320660+0000 '>' 2024-06-11T17:35:45.534568+0000 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:36:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 165073"' 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 165073' 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 165075"' 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 165075' 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 165073 165075' 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 165073 2024-06-11T17:36:22.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/165076: /' 2024-06-11T17:36:22.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:36:22.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/165078: /' 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:36:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:36:23.337 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: _ 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: hinfo_key 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: snapset 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:36:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:36:23.341 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:36:23.341 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:36:23.341 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:36:23.341 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:23.342 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:36:23.343 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:36:23.344 INFO:tasks.workunit.client.0.smithi012.stderr:165076: start osd.3 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:36:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:36:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: _ 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: hinfo_key 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: snapset 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:36:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:36:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:36:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:36:23.368 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:36:23.369 INFO:tasks.workunit.client.0.smithi012.stderr:165078: start osd.1 2024-06-11T17:36:27.913 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:36:27.913 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:36:27.913 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:36:27.913 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2024-06-11T17:36:23.385+0000 7f3a1273d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2024-06-11T17:36:23.409+0000 7f3a1273d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2024-06-11T17:36:23.413+0000 7f3a1273d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 0 2024-06-11T17:36:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2024-06-11T17:36:23.985+0000 7f3a1273d7c0 -1 Falling back to public interface 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 1 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:36:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2024-06-11T17:36:27.649+0000 7f3a1273d7c0 -1 osd.3 89 log_to_monitors true 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 3 2024-06-11T17:36:27.916 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2024-06-11T17:36:23.397+0000 7f0353d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:28.377 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2024-06-11T17:36:23.409+0000 7f0353d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2024-06-11T17:36:23.413+0000 7f0353d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 0 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2024-06-11T17:36:23.993+0000 7f0353d297c0 -1 Falling back to public interface 2024-06-11T17:36:28.378 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 1 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:28.379 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2024-06-11T17:36:27.617+0000 7f0353d297c0 -1 osd.1 89 log_to_monitors true 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 3 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:30.641 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 4 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: osd.3 up in weight 1 up_from 94 up_thru 94 down_at 90 last_clean_interval [84,89) [v2:127.0.0.1:6810/2409946900,v1:127.0.0.1:6811/2409946900] [v2:127.0.0.1:6812/2409946900,v1:127.0.0.1:6813/2409946900] exists,up bcef9271-fd5d-48a0-8a58-99f9612def0d 2024-06-11T17:36:30.642 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:36:30.643 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 1 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 2 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: 3' 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:30.644 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836549 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836549 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836549' 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperseph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 4 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: osd.1 up in weight 1 up_from 94 up_thru 94 down_at 90 last_clean_interval [87,89) [v2:127.0.0.1:6826/2067665997,v1:127.0.0.1:6827/2067665997] [v2:127.0.0.1:6828/2067665997,v1:127.0.0.1:6829/2067665997] exists,up b2d8681e-dbb3-48ae-bf03-7e16e7c4930a 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:36:30.645 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:36:30.646 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 1 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 2 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: 3' 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836550 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836550 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550' 2024-06-11T17:36:30.647 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925827 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925827 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-403726925827' 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449100 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449100 2024-06-11T17:36:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-403726925827 2-335007449100' 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925826 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925826 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-403726925827 2-335007449100 3-403726925826' 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836550 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836550 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836550 2024-06-11T17:36:32.722 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: waiting osd.0 seq 21474836550 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836548 -lt 21474836550 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836550 -lt 21474836550 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-403726925827 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-403726925827 2024-06-11T17:36:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:16507.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925826 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925826 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836549 1-403726925826' 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449101 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449101 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836549 1-403726925826 2-335007449101' 2024-06-11T17:36:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925827 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925827 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836549 1-403726925826 2-335007449101 3-403726925827' 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836549 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836549 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836549 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: waiting osd.0 seq 21474836549 2024-06-11T17:36:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836548 -lt 21474836549 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836550 -lt 21474836549 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-403726925826 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:36:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-403726925826 2024-06-11T17:36:34.136 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ub8: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925827 2024-06-11T17:36:34.136 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: waiting osd.1 seq 403726925827 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925827 -lt 403726925827 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449100 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449100 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449100 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449100' 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: waiting osd.2 seq 335007449100 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449101 -lt 335007449100 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-403726925826 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-403726925826 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925826 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 403726925826' 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: waiting osd.3 seq 403726925826 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925827 -lt 403726925826 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:34.138 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925826 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: waiting osd.1 seq 403726925826 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925827 -lt 403726925826 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449101 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449101 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449101 2024-06-11T17:36:34.146 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: waiting osd.2 seq 335007449101 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449101 -lt 335007449101 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-403726925827 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-403726925827 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925827 2024-06-11T17:36:34.147 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 403726925827' 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: waiting osd.3 seq 403726925827 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925827 -lt 403726925827 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:36:34.148 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:165076: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:34.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:34.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:36:34.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 165075 2024-06-11T17:36:34.882 INFO:tasks.workunit.client.0.smithi012.stderr:contains("stale") | not)' 2024-06-11T17:36:34.882 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:36:34.882 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:165078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:36:34.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:36:34.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:36:34.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:36:34.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:36:34.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:36:34.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:36:34.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:36:34.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:34.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:35.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:35.231 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:36:35.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:36:35.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:36:35.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:36:35.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:36:35.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:36:35.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:36:35.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:36:35.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:36:35.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:36:35.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:36:35.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:36:35.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:36:35.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:36:35.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:35.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:35.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:36:35.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:36:35.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:36:35.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:36:35.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:36:35.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:36:35.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:36:35.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:36:35.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:36:35.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:36:35.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:36:35.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:36:35.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:36:35.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:36:35.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:36:35.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:36:35.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:36:35.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:36:35.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:36:35.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:36:35.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:36:35.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:36:35.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:36:35.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:36:35.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:35.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:35.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:36:35.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:36:35.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:36:35.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:36:35.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:36:35.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:35.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:35.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:36:35.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-11T17:36:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:36:35.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:36:35.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:36:35.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:35.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:35.834 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:35.834 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:35.834 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:35.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:35.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:36:36.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:36:36.103 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:36:36.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:36:36.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:36:36.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:36:36.104 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:36:36.104 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:36:36.105 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:36:36.105 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:36:36.105 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:36:36.105 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:36.106 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:36.106 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:36:36.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:36:36.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:36:36.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:36:36.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:36:36.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:36:36.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:36:36.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:36:36.226 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:36:36.226 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:36:36.226 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:36:36.227 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:36:36.227 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:36.227 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:36.227 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:36:36.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:36:36.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:36:36.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:36:37.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:36:37.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:37.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:36:37.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:36:37.294 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:36:37.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:37.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:36:37.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:36:37.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c7583494-43a1-4b65-b8f8-4c91023fe7ba 2024-06-11T17:36:37.301 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 c7583494-43a1-4b65-b8f8-4c91023fe7ba 2024-06-11T17:36:37.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c7583494-43a1-4b65-b8f8-4c91023fe7ba' 2024-06-11T17:36:37.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:36:37.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAli2hmU33uEhAA57TlOOiWHXoBSpIqScpSyw== 2024-06-11T17:36:37.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAli2hmU33uEhAA57TlOOiWHXoBSpIqScpSyw=="}' 2024-06-11T17:36:37.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c7583494-43a1-4b65-b8f8-4c91023fe7ba -i td/osd-scrub-repair/0/new.json 2024-06-11T17:36:37.630 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:36:37.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:36:37.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAli2hmU33uEhAA57TlOOiWHXoBSpIqScpSyw== --osd-uuid c7583494-43a1-4b65-b8f8-4c91023fe7ba 2024-06-11T17:36:37.681 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.677+0000 7fa3e66117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:37.714 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.713+0000 7fa3e66117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.713+0000 7fa3e66117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:37.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.713+0000 7fa3e66117c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:36:37.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.713+0000 7fa3e66117c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:36:37.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.713+0000 7fa3e66117c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:36:37.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:37.821+0000 7fa3e66117c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:36:45.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:36:45.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:36:45.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:36:45.309 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:36:45.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:36:45.722 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:36:45.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:36:45.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:36:45.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:36:45.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:36:45.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:36:45.758 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:45.757+0000 7f50732177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:45.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:45.801+0000 7f50732177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:45.806 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:45.805+0000 7f50732177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:36:46.028 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:36:46.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:46.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:46.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:46.389+0000 7f50732177c0 -1 Falling back to public interface 2024-06-11T17:36:47.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:47.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:47.349 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:36:47.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:36:47.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:47.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:47.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:48.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:48.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:48.676 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:36:48.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:36:48.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:48.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:49.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:49.165 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:36:49.161+0000 7f50732177c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:36:50.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:50.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:50.010 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:36:50.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:36:50.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:50.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:50.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:51.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:51.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:51.366 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:36:51.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:36:51.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:51.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:51.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:52.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:52.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:52.704 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:36:52.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:36:52.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:52.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:53.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:54.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:54.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:54.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:36:54.049 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:36:54.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:54.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:54.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:55.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:55.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:55.482 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:36:55.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:36:55.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:55.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:55.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:56.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:56.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:56.823 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:36:56.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:36:56.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:56.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:57.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:58.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:58.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:58.277 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:36:58.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:36:58.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:58.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:36:58.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:36:59.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:36:59.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:36:59.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:36:59.620 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:36:59.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:36:59.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:00.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:01.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:01.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:01.049 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:37:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:37:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:01.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:02.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:02.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:02.529 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:37:02.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:37:02.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:02.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:03.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:04.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:04.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:04.071 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:37:04.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:37:04.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:04.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:04.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:05.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:05.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:05.499 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:37:05.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:37:05.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:05.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:05.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:06.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:06.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:06.839 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:37:06.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:37:06.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:06.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:07.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:08.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:08.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:08.377 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:37:08.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:37:08.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:08.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:09.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:10.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:10.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:10.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:37:10.320 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:37:10.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:10.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:10.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:11.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:11.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:11.956 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:37:11.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:37:11.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:11.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:13.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:13.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:13.567 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:37:13.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:37:13.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:13.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:13.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:14.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:14.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:14.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:37:14.986 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:37:14.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:14.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:15.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:16.883 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:37:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:37:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:17.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:18.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:18.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:18.455 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:37:18.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:37:18.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:18.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:19.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:20.121 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:37:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:37:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:37:20.464 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1084342734,v1:127.0.0.1:6803/1084342734] [v2:127.0.0.1:6804/1084342734,v1:127.0.0.1:6805/1084342734] exists,up c7583494-43a1-4b65-b8f8-4c91023fe7ba 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:37:20.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:37:20.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:37:20.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:37:20.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:37:20.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:37:20.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:37:20.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 41c497b1-27ae-4b67-8d11-012ee9169fff' 2024-06-11T17:37:20.472 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:37:20.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:37:20.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQi2hmESIqHRAAbFsS/fdf1QMAXX20izBBUg== 2024-06-11T17:37:20.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQi2hmESIqHRAAbFsS/fdf1QMAXX20izBBUg=="}' 2024-06-11T17:37:20.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41c497b1-27ae-4b67-8d11-012ee9169fff -i td/osd-scrub-repair/1/new.json 2024-06-11T17:37:20.891 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:37:20.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:37:20.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQi2hmESIqHRAAbFsS/fdf1QMAXX20izBBUg== --osd-uuid 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:37:20.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:20.941+0000 7efc2ee507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:21.010 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.009+0000 7efc2ee507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:21.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.009+0000 7efc2ee507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:21.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.009+0000 7efc2ee507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:37:21.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.009+0000 7efc2ee507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:37:21.012 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.009+0000 7efc2ee507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:37:21.097 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:21.093+0000 7efc2ee507c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:37:29.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:37:29.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:37:29.406 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:37:29.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:37:29.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:37:29.826 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:37:29.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:37:29.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:37:29.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:37:29.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:37:29.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:37:29.860 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:29.857+0000 7f83706c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:29.899 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:29.897+0000 7f83706c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:29.901 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:29.901+0000 7f83706c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:37:30.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:30.195 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:37:30.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:37:30.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:30.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:30.464 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:30.461+0000 7f83706c87c0 -1 Falling back to public interface 2024-06-11T17:37:30.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:31.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:31.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:31.607 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:37:31.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:37:31.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:31.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:31.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:32.932 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:37:32.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:37:32.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:32.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:33.230 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:37:33.229+0000 7f83706c87c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:37:33.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:34.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:34.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:34.279 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:37:34.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:37:34.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:34.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:34.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:35.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:35.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:35.872 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:37:35.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:37:35.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:35.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:36.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:37.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:37.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:37.386 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:37:37.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:37:37.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:37.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:37.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:38.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:38.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:38.726 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:37:38.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:37:38.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:38.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:39.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:40.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:40.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:40.231 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:37:40.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:37:40.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:40.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:40.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:41.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:41.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:41.646 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:37:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:37:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:42.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:43.062 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:37:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:37:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:43.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:44.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:44.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:44.544 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:37:44.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:37:44.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:44.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:45.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:45.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:45.919 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:37:45.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:37:45.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:45.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:46.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:47.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:47.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:47.302 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:37:47.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:37:47.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:47.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:47.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:48.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:48.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:48.632 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:37:48.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:37:48.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:48.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:49.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:50.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:50.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:50.295 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:37:50.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:37:50.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:50.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:50.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:51.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:51.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:51.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:37:51.711 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:37:51.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:51.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:52.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:53.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:53.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:53.131 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:37:53.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:37:53.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:53.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:53.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:37:54.808 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:37:54.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:54.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:55.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:56.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:56.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:56.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:37:56.811 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:37:56.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:56.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:37:58.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:37:58.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:37:58.553 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:37:58.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:37:58.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:37:58.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:37:59.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:00.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:00.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:00.450 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:38:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:38:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:38:01.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:02.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:02.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:02.496 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:38:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:38:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:02.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:38:03.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:04.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:04.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:04.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:38:04.400 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:38:04.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:04.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:38:04.782 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2416177499,v1:127.0.0.1:6811/2416177499] [v2:127.0.0.1:6812/2416177499,v1:127.0.0.1:6813/2416177499] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:38:04.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:38:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:38:04.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:38:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:38:04.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:38:04.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:38:04.789 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:38:04.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa' 2024-06-11T17:38:04.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:38:04.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8i2hm2WIpMBAAnJRXWh330ql97tgMBcVwqg== 2024-06-11T17:38:04.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8i2hm2WIpMBAAnJRXWh330ql97tgMBcVwqg=="}' 2024-06-11T17:38:04.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa -i td/osd-scrub-repair/2/new.json 2024-06-11T17:38:05.207 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:38:05.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:38:05.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8i2hm2WIpMBAAnJRXWh330ql97tgMBcVwqg== --osd-uuid b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:38:05.260 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.257+0000 7f7509ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:05.358 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.357+0000 7f7509ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:05.361 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.357+0000 7f7509ae07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:05.361 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.357+0000 7f7509ae07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:38:05.361 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.357+0000 7f7509ae07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:38:05.361 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.357+0000 7f7509ae07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:38:05.447 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:05.445+0000 7f7509ae07c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:38:13.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:38:13.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:38:13.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:38:13.681 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:38:13.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:38:14.085 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:38:14.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:38:14.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:38:14.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:38:14.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:38:14.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:38:14.118 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:14.116+0000 7f114d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:14.164 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:14.160+0000 7f114d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:14.166 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:14.164+0000 7f114d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:38:14.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:14.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:38:14.397 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:38:14.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:14.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:14.757 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:14.752+0000 7f114d9157c0 -1 Falling back to public interface 2024-06-11T17:38:15.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:15.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:15.711 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:38:15.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:38:15.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:15.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:17.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:17.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:17.043 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:38:17.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:38:17.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:17.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:17.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:17.544 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:17.540+0000 7f114d9157c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:38:18.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:18.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:18.380 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:38:18.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:38:18.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:18.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:18.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:19.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:19.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:19.935 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:38:19.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:38:19.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:19.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:20.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:21.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:21.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:21.269 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:38:21.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:38:21.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:21.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:21.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:22.741 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:38:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:38:22.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:22.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:23.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:24.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:24.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:24.076 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:38:24.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:38:24.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:24.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:24.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:25.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:25.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:25.412 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:38:25.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:38:25.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:25.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:25.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:26.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:26.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:26.903 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:38:26.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:38:26.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:26.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:27.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:28.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:28.237 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:38:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:38:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:28.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:29.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:29.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:29.574 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:38:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:38:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:29.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:30.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:30.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:30.975 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:38:30.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:38:30.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:30.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:31.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:32.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:32.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:32.312 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:38:32.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:38:32.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:32.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:33.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:33.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:33.654 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:38:33.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:38:33.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:33.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:33.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:34.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:34.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:34.995 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:38:34.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:38:34.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:34.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:35.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:36.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:36.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:36.329 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:38:36.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:38:36.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:36.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:36.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:37.839 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:38:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:38:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:38.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:39.473 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:38:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:38:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:40.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:41.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:41.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:41.150 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:38:41.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:38:41.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:41.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:42.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:42.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:42.527 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:38:42.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:38:42.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:42.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:43.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:44.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:44.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:44.464 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:38:44.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:38:44.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:44.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:45.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:46.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:46.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:46.143 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:38:46.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:38:46.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:46.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:46.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:47.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:47.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:47.766 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:38:47.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:38:47.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:47.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:38:48.122 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4276704457,v1:127.0.0.1:6819/4276704457] [v2:127.0.0.1:6820/4276704457,v1:127.0.0.1:6821/4276704457] exists,up b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:38:48.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:38:48.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:38:48.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:38:48.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:38:48.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:38:48.126 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:38:48.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:38:48.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:38:48.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:38:48.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:38:48.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:38:48.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:38:48.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1d37955f-ed1a-433e-abb1-df64da9a5f53' 2024-06-11T17:38:48.129 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:38:48.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:38:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoi2hmKxPLCBAAIgDekJSW06/pB7LGo/gbsA== 2024-06-11T17:38:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoi2hmKxPLCBAAIgDekJSW06/pB7LGo/gbsA=="}' 2024-06-11T17:38:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d37955f-ed1a-433e-abb1-df64da9a5f53 -i td/osd-scrub-repair/3/new.json 2024-06-11T17:38:48.735 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:38:48.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-11T17:38:48.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoi2hmKxPLCBAAIgDekJSW06/pB7LGo/gbsA== --osd-uuid 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:38:48.794 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.792+0000 7fb21717d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:48.862 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.860+0000 7fb21717d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:48.864 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.860+0000 7fb21717d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:48.864 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.860+0000 7fb21717d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:38:48.864 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.860+0000 7fb21717d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:38:48.865 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.860+0000 7fb21717d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:38:48.950 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:48.948+0000 7fb21717d7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-11T17:38:57.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-11T17:38:57.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:38:57.276 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:38:57.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:38:57.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:38:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:38:57.694 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:38:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:38:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:38:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:38:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:38:57.730 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:57.728+0000 7fcb7cd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:57.764+0000 7fcb7cd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:57.768+0000 7fcb7cd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:58.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:38:58.012 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:38:58.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:58.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:38:58.335 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:38:58.332+0000 7fcb7cd897c0 -1 Falling back to public interface 2024-06-11T17:38:58.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:38:59.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:38:59.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:38:59.407 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:38:59.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:38:59.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:38:59.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:38:59.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:00.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:00.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:00.747 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:39:00.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:39:00.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:00.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:01.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:01.358 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:39:01.356+0000 7fcb7cd897c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:39:02.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:02.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:02.083 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:39:02.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:39:02.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:02.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:02.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:03.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:03.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:03.567 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:39:03.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:39:03.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:03.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:03.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:39:04.983 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:39:04.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:04.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:05.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:06.486 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:39:06.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:39:06.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:06.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:06.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:07.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:07.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:07.865 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:39:07.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:39:07.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:07.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:08.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:09.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:09.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:09.244 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:39:09.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:39:09.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:09.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:09.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:10.579 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:39:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:39:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:11.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:12.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:12.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:12.137 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:39:12.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:39:12.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:12.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:12.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:13.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:13.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:13.666 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:39:13.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:39:13.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:13.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:13.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:14.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:14.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:14.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:39:14.999 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:39:15.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:15.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:15.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:16.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:16.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:16.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:39:16.498 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:39:16.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:16.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:16.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:17.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:17.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:17.861 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:39:17.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:39:17.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:17.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:18.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:19.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:19.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:19.275 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:39:19.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:39:19.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:19.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:19.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:20.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:20.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:20.616 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:39:20.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:39:20.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:20.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:21.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:22.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:22.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:22.721 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:39:22.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:39:22.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:22.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:23.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:24.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:24.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:24.684 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:39:24.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:39:24.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:24.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:25.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:26.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:26.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:26.347 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:39:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:39:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:26.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:27.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:27.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:27.759 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:39:27.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:39:27.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:27.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:28.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:29.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:29.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:29.563 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:39:29.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:39:29.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:29.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:30.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:31.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:31.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:31.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:39:31.023 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:39:31.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:31.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:39:32.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:39:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:32.611 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:39:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:39:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:32.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:32.946 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3327047873,v1:127.0.0.1:6827/3327047873] [v2:127.0.0.1:6828/3327047873,v1:127.0.0.1:6829/3327047873] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:39:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:39:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:39:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:39:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-11T17:39:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:39:33.306 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:39:33.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:39:33.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:39:33.759 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:39:33.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:39:34.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:39:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-11T17:39:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:39:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:39:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:39:38.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:39:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:39:38.408 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:39:38.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:39:38.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:39:38.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:39:38.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:39:38.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:39:38.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:39:38.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:39:38.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:39:38.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:39:39.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:39:39.078 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:39:39.079 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:39:39.079 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:39:39.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:39:39.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:39.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:39:39.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836515 2024-06-11T17:39:39.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836515 2024-06-11T17:39:39.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515' 2024-06-11T17:39:39.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:39.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:39:39.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672986 2024-06-11T17:39:39.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672986 2024-06-11T17:39:39.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672986' 2024-06-11T17:39:39.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:39.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:39:39.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-06-11T17:39:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-06-11T17:39:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509454' 2024-06-11T17:39:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:39.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:39:39.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-06-11T17:39:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-06-11T17:39:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509454 3-85899345932' 2024-06-11T17:39:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:39.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836515 2024-06-11T17:39:39.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:39:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-06-11T17:39:39.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:39.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836515 2024-06-11T17:39:39.729 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836515 2024-06-11T17:39:39.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-06-11T17:39:39.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:39:40.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836515 2024-06-11T17:39:40.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:39:41.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:39:41.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:39:41.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836515 2024-06-11T17:39:41.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:41.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672986 2024-06-11T17:39:41.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:41.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:39:41.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-06-11T17:39:41.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:41.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672986 2024-06-11T17:39:41.404 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672986 2024-06-11T17:39:41.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-11T17:39:41.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:39:41.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672987 -lt 42949672986 2024-06-11T17:39:41.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:41.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-06-11T17:39:41.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:41.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:39:41.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-06-11T17:39:41.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:41.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-06-11T17:39:41.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-11T17:39:41.778 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509454 2024-06-11T17:39:41.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:39:42.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-11T17:39:42.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:42.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345932 2024-06-11T17:39:42.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:42.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:39:42.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345932 2024-06-11T17:39:42.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:42.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-06-11T17:39:42.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-06-11T17:39:42.169 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345932 2024-06-11T17:39:42.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:39:42.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-06-11T17:39:42.582 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:39:42.582 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:39:42.583 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:39:43.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:39:43.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:39:43.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:39:43.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:39:43.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:39:43.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:39:43.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:39:43.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:39:43.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:39:43.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:39:43.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-11T17:39:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:39:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:39:45.467 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:39:45.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:39:46.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-06-11T17:39:46.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-11T17:39:47.032 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 2 allow_ec_overwrites to true 2024-06-11T17:39:47.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:39:47.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:39:47.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:39:47.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:39:47.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:39:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:39:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:39:47.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:39:47.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:39:47.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:39:47.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:47.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:39:47.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836518 2024-06-11T17:39:47.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836518 2024-06-11T17:39:47.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518' 2024-06-11T17:39:47.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:47.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:39:48.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-06-11T17:39:48.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-06-11T17:39:48.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672989' 2024-06-11T17:39:48.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:48.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:39:48.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-06-11T17:39:48.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-06-11T17:39:48.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672989 2-64424509457' 2024-06-11T17:39:48.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:39:48.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:39:48.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345935 2024-06-11T17:39:48.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345935 2024-06-11T17:39:48.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672989 2-64424509457 3-85899345935' 2024-06-11T17:39:48.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:48.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836518 2024-06-11T17:39:48.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:48.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:39:48.572 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:48.572 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836518 2024-06-11T17:39:48.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836518 2024-06-11T17:39:48.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-06-11T17:39:48.573 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836518 2024-06-11T17:39:48.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:39:48.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836518 -lt 21474836518 2024-06-11T17:39:48.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:48.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:48.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-06-11T17:39:48.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:39:48.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-06-11T17:39:48.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:48.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-06-11T17:39:48.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-11T17:39:48.894 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672989 2024-06-11T17:39:48.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:39:49.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-11T17:39:49.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:49.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-06-11T17:39:49.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:49.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:39:49.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-06-11T17:39:49.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:49.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-06-11T17:39:49.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-11T17:39:49.262 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509457 2024-06-11T17:39:49.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:39:49.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509457 2024-06-11T17:39:49.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:39:50.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:39:50.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:39:50.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-11T17:39:50.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:39:50.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:39:50.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345935 2024-06-11T17:39:50.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:39:50.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345935 2024-06-11T17:39:50.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:39:50.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345935 2024-06-11T17:39:50.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2024-06-11T17:39:50.947 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345935 2024-06-11T17:39:50.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:39:51.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345935 -lt 85899345935 2024-06-11T17:39:51.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:39:51.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:39:51.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:39:51.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:39:51.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:39:51.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:39:51.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:39:51.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:39:51.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:39:51.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:39:51.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:39:52.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:39:52.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:39:52.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:39:52.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-11T17:39:52.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T17:39:52.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T17:39:52.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T17:39:52.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T17:39:52.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T17:39:52.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T17:39:52.971 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:39:52.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T17:39:53.516 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:39:53.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T17:39:53.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T17:39:53.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:39:53.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-11T17:39:53.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:39:53.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:39:53.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:39:53.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:39:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-11T17:39:54.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-11T17:39:54.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T17:39:54.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-11T17:39:54.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-11T17:39:54.276 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:39:54.276 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T17:39:54.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-11T17:39:54.645 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:39:54.645 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:39:54.645 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T17:39:54.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:39:54.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:39:54.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:39:54.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:39:54.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:39:54.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:39:54.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:39:55.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:39:55.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:39:55.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:39:55.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:39:55.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:39:55.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:39:55.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:39:55.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:39:57.802 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:39:58.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:39:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:39:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:39:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:39:58.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:39:58.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:39:58.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:39:58.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:39:58.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:39:58.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:39:58.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:39:58.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:39:58.341 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:39:58.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:39:58.342 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:39:58.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:39:58.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:39:58.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:39:58.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:39:58.377 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:39:58.376+0000 7f9b974897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:39:58.380 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:39:58.376+0000 7f9b974897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:39:58.381 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:39:58.380+0000 7f9b974897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:39:58.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:39:58.682 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:39:58.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:39:58.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:39:58.908 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:39:58.904+0000 7f9b974897c0 -1 Falling back to public interface 2024-06-11T17:39:59.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:00.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:00.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:00.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:40:00.042 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:00.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:00.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:00.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:01.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:40:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:01.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:01.733 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:01.732+0000 7f9b974897c0 -1 osd.3 39 log_to_monitors true 2024-06-11T17:40:02.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:02.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:40:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:40:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 43 up_thru 34 down_at 40 last_clean_interval [20,39) [v2:127.0.0.1:6826/2253796787,v1:127.0.0.1:6827/2253796787] [v2:127.0.0.1:6828/2253796787,v1:127.0.0.1:6829/2253796787] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:40:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:40:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:40:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:40:03.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:40:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:40:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:40:03.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:40:03.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:40:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:40:03.175 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:40:03.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:40:03.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:40:03.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:40:03.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:03.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:40:04.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836522 2024-06-11T17:40:04.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836522 2024-06-11T17:40:04.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522' 2024-06-11T17:40:04.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:04.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:40:04.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672993 2024-06-11T17:40:04.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672993 2024-06-11T17:40:04.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672993' 2024-06-11T17:40:04.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:04.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:40:04.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509461 2024-06-11T17:40:04.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509461 2024-06-11T17:40:04.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672993 2-64424509461' 2024-06-11T17:40:04.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:04.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:40:04.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=184683593730 2024-06-11T17:40:04.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 184683593730 2024-06-11T17:40:04.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672993 2-64424509461 3-184683593730' 2024-06-11T17:40:04.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:04.879 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836522 2024-06-11T17:40:04.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:04.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:40:04.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836522 2024-06-11T17:40:04.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836522 2024-06-11T17:40:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-06-11T17:40:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836522 2024-06-11T17:40:04.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:05.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836522 -lt 21474836522 2024-06-11T17:40:05.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:05.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672993 2024-06-11T17:40:05.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:05.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:40:05.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672993 2024-06-11T17:40:05.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:05.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672993 2024-06-11T17:40:05.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-06-11T17:40:05.229 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672993 2024-06-11T17:40:05.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:40:05.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672992 -lt 42949672993 2024-06-11T17:40:05.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:40:06.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:40:06.595 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:40:06.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672994 -lt 42949672993 2024-06-11T17:40:06.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:06.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509461 2024-06-11T17:40:06.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:06.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:40:06.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509461 2024-06-11T17:40:06.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:06.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509461 2024-06-11T17:40:06.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-11T17:40:06.936 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509461 2024-06-11T17:40:06.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:40:07.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-11T17:40:07.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:07.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-184683593730 2024-06-11T17:40:07.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:07.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:40:07.314 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-184683593730 2024-06-11T17:40:07.315 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=184683593730 2024-06-11T17:40:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-06-11T17:40:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 184683593730 2024-06-11T17:40:07.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:40:07.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593730 -lt 184683593730 2024-06-11T17:40:07.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:40:07.690 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:07.690 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:08.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:40:08.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:40:08.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:40:08.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:40:08.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:40:08.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:40:08.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:40:08.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:40:08.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:40:08.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:40:08.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:08.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:40:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:40:08.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:40:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:40:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:40:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:40:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:40:08.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:40:09.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:40:09.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:40:09.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:40:09.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:40:09.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:09.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:09.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:09.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:09.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:09.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:40:10.001 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:40:10.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:10.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:40:10.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:10.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:40:10.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:40:10.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:10.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:10.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:10.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:10.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:10.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:10.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:39:45.300297+0000 '>' 2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:10.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:40:11.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:40:11.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:11.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:11.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:11.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:11.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:11.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:11.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:39:45.300297+0000 '>' 2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:11.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:40:12.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:40:12.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:12.674 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:12.674 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:12.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:12.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:12.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:13.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:10.589491+0000 '>' 2024-06-11T17:39:45.300297+0000 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:40:13.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:40:13.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:40:13.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:40:13.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:40:13.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:40:13.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:13.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:40:13.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:40:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:40:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:40:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:40:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:40:13.460 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:40:13.460 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:40:13.460 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:40:13.460 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:40:13.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:40:13.750 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:40:13.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:40:13.753 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:40:13.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:40:13.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:40:13.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:40:13.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:40:13.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:40:13.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:40:13.789 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:13.784+0000 7f9376f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:13.808+0000 7f9376f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:13.812 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:13.808+0000 7f9376f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:14.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:14.348+0000 7f9376f8d7c0 -1 Falling back to public interface 2024-06-11T17:40:14.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:14.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:15.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:40:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:16.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:16.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:17.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:17.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:17.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:40:17.486 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:17.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:17.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:17.652+0000 7f9376f8d7c0 -1 osd.3 45 log_to_monitors true 2024-06-11T17:40:17.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:18.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:18.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:18.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:40:18.823 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:40:18.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:18.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:19.563 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/272602132,v1:127.0.0.1:6827/272602132] [v2:127.0.0.1:6828/272602132,v1:127.0.0.1:6829/272602132] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:40:19.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:40:19.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:40:19.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:40:20.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:20.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:40:20.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-06-11T17:40:20.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-06-11T17:40:20.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-06-11T17:40:20.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:20.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:40:20.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-06-11T17:40:20.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-06-11T17:40:20.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949672998' 2024-06-11T17:40:20.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:20.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:40:20.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509465 2024-06-11T17:40:20.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509465 2024-06-11T17:40:20.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949672998 2-64424509465' 2024-06-11T17:40:20.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:20.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:40:20.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364802 2024-06-11T17:40:20.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364802 2024-06-11T17:40:20.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949672998 2-64424509465 3-214748364802' 2024-06-11T17:40:20.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:20.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-06-11T17:40:20.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:20.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:40:20.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-06-11T17:40:20.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:20.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-06-11T17:40:20.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-06-11T17:40:20.992 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836526 2024-06-11T17:40:20.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:21.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836526 2024-06-11T17:40:21.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:40:22.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:40:22.317 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:22.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836526 2024-06-11T17:40:22.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:22.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:22.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-06-11T17:40:22.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:40:22.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-06-11T17:40:22.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:22.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-06-11T17:40:22.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-06-11T17:40:22.667 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672998 2024-06-11T17:40:22.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:40:23.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-06-11T17:40:23.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:23.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509465 2024-06-11T17:40:23.035 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:23.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:40:23.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509465 2024-06-11T17:40:23.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:23.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509465 2024-06-11T17:40:23.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-06-11T17:40:23.039 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509465 2024-06-11T17:40:23.039 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:40:23.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509465 -lt 64424509465 2024-06-11T17:40:23.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:23.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-214748364802 2024-06-11T17:40:23.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:23.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:40:23.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-214748364802 2024-06-11T17:40:23.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364802 2024-06-11T17:40:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-06-11T17:40:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 214748364802 2024-06-11T17:40:23.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:40:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364802 -lt 214748364802 2024-06-11T17:40:23.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:40:23.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:23.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:24.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:40:24.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:40:24.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:40:24.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:40:24.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:40:24.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:40:24.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:40:24.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:40:24.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:40:24.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:40:24.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:24.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:40:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:40:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:40:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:40:24.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:40:24.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-11T17:40:24.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:40:24.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:40:24.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:40:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:40:28.150 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:40:28.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:40:28.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:40:28.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:40:28.687 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:40:28.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:40:28.690 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T17:40:28.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:40:28.691 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:40:28.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:40:28.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:40:28.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:40:28.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:40:28.730 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:28.727+0000 7f12266f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:28.733 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:28.731+0000 7f12266f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:28.734 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:28.731+0000 7f12266f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:29.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:40:29.272 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:29.267+0000 7f12266f67c0 -1 Falling back to public interface 2024-06-11T17:40:29.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:30.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:30.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:30.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:40:30.403 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:30.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:30.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:40:30.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:31.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:31.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:31.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:40:31.764 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:31.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:31.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:40:32.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:32.205 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:32.187+0000 7f12266f67c0 -1 osd.1 52 log_to_monitors true 2024-06-11T17:40:33.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:33.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:33.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:40:33.105 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:40:33.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:33.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/2634888316,v1:127.0.0.1:6811/2634888316] [v2:127.0.0.1:6812/2634888316,v1:127.0.0.1:6813/2634888316] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:40:33.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:40:33.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:40:33.696 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:40:33.696 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:40:33.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:40:33.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:40:33.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:40:33.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:40:34.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:34.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:40:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-06-11T17:40:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-06-11T17:40:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-06-11T17:40:34.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:34.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:40:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135874 2024-06-11T17:40:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135874 2024-06-11T17:40:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-244813135874' 2024-06-11T17:40:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:34.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:40:35.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-06-11T17:40:35.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-06-11T17:40:35.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-244813135874 2-64424509469' 2024-06-11T17:40:35.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:35.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364806 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364806 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-244813135874 2-64424509469 3-214748364806' 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:35.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-06-11T17:40:35.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:40:35.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-06-11T17:40:35.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:35.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-06-11T17:40:35.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-06-11T17:40:35.219 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836530 2024-06-11T17:40:35.219 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:35.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836529 -lt 21474836530 2024-06-11T17:40:35.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:40:36.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:40:36.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:36.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836531 -lt 21474836530 2024-06-11T17:40:36.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:36.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135874 2024-06-11T17:40:36.882 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:36.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:40:36.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135874 2024-06-11T17:40:36.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:36.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135874 2024-06-11T17:40:36.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-06-11T17:40:36.887 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135874 2024-06-11T17:40:36.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:40:37.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135874 -lt 244813135874 2024-06-11T17:40:37.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:37.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-06-11T17:40:37.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:37.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:40:37.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-06-11T17:40:37.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:37.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-06-11T17:40:37.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-06-11T17:40:37.288 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509469 2024-06-11T17:40:37.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:40:37.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-06-11T17:40:37.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:37.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-214748364806 2024-06-11T17:40:37.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:37.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:40:37.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-214748364806 2024-06-11T17:40:37.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:37.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364806 2024-06-11T17:40:37.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-06-11T17:40:37.666 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 214748364806 2024-06-11T17:40:37.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:40:37.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364806 -lt 214748364806 2024-06-11T17:40:37.985 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:40:37.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:37.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:38.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:40:38.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:40:38.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:40:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:40:38.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:40:38.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:38.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:39.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:40:39.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:40:39.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:40:39.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:40:39.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:40:39.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:40:39.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:40:39.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:40:39.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:40:39.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:40:39.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:40:39.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:40:39.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:39.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:39.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:39.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:39.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:39.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:40:40.209 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:40:40.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:40.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:40:40.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:40.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:40:40.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:40:40.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:40.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:40.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:40.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:40.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:40.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:40.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:10.589491+0000 '>' 2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:40.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:40:41.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:40:41.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:41.563 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:41.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:41.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:41.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:41.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:41.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:10.589491+0000 '>' 2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:41.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:40:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:40:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:42.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:42.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:42.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:42.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:42.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:43.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:10.589491+0000 '>' 2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:43.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:40:44.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:40:44.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:40:44.223 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:40:44.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:40:44.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:40:44.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:40:44.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:41.093720+0000 '>' 2024-06-11T17:40:10.589491+0000 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:40:44.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:40:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:40:45.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:40:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:40:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:40:46.194 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:40:46.194 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:40:46.194 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:40:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:40:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:40:46.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:46.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:40:46.504 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:40:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:40:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:40:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:40:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:40:46.539 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:46.535+0000 7f9874a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:46.623 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:46.619+0000 7f9874a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:46.625 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:46.619+0000 7f9874a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:40:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:40:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:40:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:40:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:46.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:47.167 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:47.163+0000 7f9874a9a7c0 -1 Falling back to public interface 2024-06-11T17:40:47.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:48.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:48.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:48.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:40:48.191 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:48.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:48.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:48.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:49.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:49.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:49.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:40:49.565 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:49.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:49.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:49.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:50.736 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:40:50.731+0000 7f9874a9a7c0 -1 osd.3 59 log_to_monitors true 2024-06-11T17:40:50.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:50.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:50.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:40:50.904 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:40:50.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:50.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:51.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:40:52.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:40:52.647 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/1328911370,v1:127.0.0.1:6827/1328911370] [v2:127.0.0.1:6828/1328911370,v1:127.0.0.1:6829/1328911370] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:40:52.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:40:52.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:40:52.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:40:52.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:40:52.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:40:52.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:40:52.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:40:52.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:40:52.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:40:52.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:40:52.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:40:52.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:40:52.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:40:52.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:40:53.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr:3' 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:53.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:40:53.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836535 2024-06-11T17:40:53.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836535 2024-06-11T17:40:53.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535' 2024-06-11T17:40:53.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:53.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:40:53.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135879 2024-06-11T17:40:53.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135879 2024-06-11T17:40:53.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-244813135879' 2024-06-11T17:40:53.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:53.509 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:40:53.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509474 2024-06-11T17:40:53.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509474 2024-06-11T17:40:53.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-244813135879 2-64424509474' 2024-06-11T17:40:53.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:40:53.657 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:40:53.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906946 2024-06-11T17:40:53.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906946 2024-06-11T17:40:53.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-244813135879 2-64424509474 3-274877906946' 2024-06-11T17:40:53.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:53.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836535 2024-06-11T17:40:53.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:53.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:40:53.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836535 2024-06-11T17:40:53.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:53.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836535 2024-06-11T17:40:53.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-06-11T17:40:53.806 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836535 2024-06-11T17:40:53.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:54.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836535 2024-06-11T17:40:54.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:40:55.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:40:55.112 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:40:55.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836535 -lt 21474836535 2024-06-11T17:40:55.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:55.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135879 2024-06-11T17:40:55.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:55.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:40:55.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135879 2024-06-11T17:40:55.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:55.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135879 2024-06-11T17:40:55.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-06-11T17:40:55.443 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135879 2024-06-11T17:40:55.443 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:40:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135879 -lt 244813135879 2024-06-11T17:40:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:55.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509474 2024-06-11T17:40:55.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:55.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:40:55.817 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509474 2024-06-11T17:40:55.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:55.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509474 2024-06-11T17:40:55.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-06-11T17:40:55.819 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509474 2024-06-11T17:40:55.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:40:56.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509474 -lt 64424509474 2024-06-11T17:40:56.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:40:56.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-274877906946 2024-06-11T17:40:56.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:40:56.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:40:56.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-274877906946 2024-06-11T17:40:56.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:40:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906946 2024-06-11T17:40:56.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-06-11T17:40:56.181 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 274877906946 2024-06-11T17:40:56.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:40:56.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906947 -lt 274877906946 2024-06-11T17:40:56.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:40:56.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:56.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:56.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:40:56.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:40:56.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:40:56.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:40:56.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:40:56.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:40:56.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:40:56.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:40:57.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:40:57.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:40:57.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:40:57.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:40:57.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:40:57.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:40:57.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:40:57.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:40:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:40:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-11T17:40:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:40:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:40:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 181452"' 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 181452' 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:40:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 181454"' 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 181454' 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/181455: /' 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 181452 181454' 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:40:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 181452 2024-06-11T17:40:57.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:40:57.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/181457: /' 2024-06-11T17:41:01.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:01.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:01.359 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:01.360 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:01.361 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:01.361 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:01.365 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: start osd.1 2024-06-11T17:41:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:01.584 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:41:01.585 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:01.586 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:01.588 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:01.588 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:01.588 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:01.589 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: start osd.2 2024-06-11T17:41:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:06.291 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2024-06-11T17:41:01.395+0000 7fbae49b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2024-06-11T17:41:01.395+0000 7fbae49b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2024-06-11T17:41:01.399+0000 7fbae49b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 0 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2024-06-11T17:41:01.939+0000 7fbae49b77c0 -1 Falling back to public interface 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 1 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.293 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2024-06-11T17:41:05.111+0000 7fbae49b77c0 -1 osd.1 66 log_to_monitors true 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 3 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.294 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:06.337 INFO:tasks.workunit.client.0.smithi012.stderr:181455: osd.1 up in weight 1 up_from 72 up_thru 60 down_at 67 last_181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T17:41:06.337 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2024-06-11T17:41:01.619+0000 7fb74eb027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2024-06-11T17:41:01.623+0000 7fb74eb027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2024-06-11T17:41:01.623+0000 7fb74eb027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:06.338 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 0 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2024-06-11T17:41:02.159+0000 7fb74eb027c0 -1 Falling back to public interface 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 1 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.339 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2024-06-11T17:41:05.683+0000 7fb74eb027c0 -1 osd.2 66 log_to_monitors true 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 3 2024-06-11T17:41:06.340 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:06.341 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [57,66) [v2:127.0.0.1:6810/4177677462,v1:127.0.0.1:6811/4177677462] [v2:127.0.0.1:6812/4177677462,v1:127.0.0.1:6813/4177677462] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:07.231 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 1 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 2 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: 3' 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:07.232 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836539 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836539 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539' 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=309237645314 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 309237645314 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-309237645314' 2024-06-11T17:41:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 4 2024-06-11T17:41:08.728 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: osd.2 up in weight 1 up_from 73 up_thru 53 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/2016641087,v1:127.0.0.1:6819/2016641087] [v2:127.0.0.1:6820/2016641087,v1:127.0.0.1:6821/2016641087] exists,up b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:08.729 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 1 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 2 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: 3' 2024-06-11T17:41:08.730 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836540 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836540 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540' 2024-06-11T17:41:08.731 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:09.944 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heltell osd.2 flush_pg_stats 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612610 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612610 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-309237645314 2-313532612610' 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906950 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906950 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836539 1-309237645314 2-313532612610 3-274877906950' 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836539 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836539 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836539 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: waiting osd.0 seq 21474836539 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836539 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836539 -lt 21474836539 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-309237645314 2024-06-11T17:41:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-309237645314 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=309237645314 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: waiting osd.1 seq 309237645314 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 309237645314 -lt 309237645314 2024-06-11T17:41:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:10.995 INFO:tasks.workunit.client.0.smithi012.stderr:1814pers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:10.995 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=309237645315 2024-06-11T17:41:10.995 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 309237645315 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-309237645315' 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612611 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612611 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-309237645315 2-313532612611' 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906951 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906951 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-309237645315 2-313532612611 3-274877906951' 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: waiting osd.0 seq 21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836539 -lt 21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836540 -lt 21474836540 2024-06-11T17:41:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-309237645315 2024-06-11T17:41:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-309237645315 2024-06-11T17:41:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/u55: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-313532612610 2024-06-11T17:41:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-313532612610 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612610 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: waiting osd.2 seq 313532612610 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612610 -lt 313532612610 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-274877906950 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-274877906950 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906950 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: waiting osd.3 seq 274877906950 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906951 -lt 274877906950 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T17:41:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:12.375 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.cbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=309237645315 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: waiting osd.1 seq 309237645315 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 309237645315 -lt 309237645315 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-313532612611 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-313532612611 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612611 2024-06-11T17:41:12.376 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: waiting osd.2 seq 313532612611 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612611 -lt 313532612611 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-274877906951 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-274877906951 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906951 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: waiting osd.3 seq 274877906951 2024-06-11T17:41:12.377 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906951 -lt 274877906951 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:12.984 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:12.984 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:12.984 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:12.985 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:41:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:41:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:41:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:41:13.106 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/stale") | not)' 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:181457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:13.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:13.901 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:181455: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:13.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 181454 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:41:13.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:41:13.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:41:13.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:41:13.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:41:13.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:41:13.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:41:14.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:41:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:41:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:41:14.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:41:14.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:14.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:14.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:14.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:14.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:40:41.093720+0000 2024-06-11T17:41:14.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:41:14.943 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:40:41.093720+0000 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:40:41.093720+0000 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:41:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:41:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:41:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:15.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:40:41.093720+0000 '>' 2024-06-11T17:40:41.093720+0000 2024-06-11T17:41:15.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:41:16.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:41:16.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:41:16.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:41:16.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:16.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:16.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:16.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:16.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:41:15.951702+0000 '>' 2024-06-11T17:40:41.093720+0000 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 184375"' 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 184375' 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:16.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 184377"' 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 184377' 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 184375 184377' 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 184375 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/184378: /' 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:41:16.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/184381: /' 2024-06-11T17:41:17.640 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:17.641 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: _ 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: hinfo_key 2024-06-11T17:41:17.642 INFO:tasks.workunit.client.0.smithi012.stderr:184378: snapset 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:17.643 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:17.645 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:17.645 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:17.645 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:41:17.645 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:17.646 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:41:17.647 INFO:tasks.workunit.client.0.smithi012.stderr:184378: start osd.1 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:17.656 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: _ 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: hinfo_key 2024-06-11T17:41:17.657 INFO:tasks.workunit.client.0.smithi012.stderr:184381: snapset 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:17.658 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:17.659 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:17.659 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:17.659 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:41:17.659 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:17.660 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T17:41:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:184381: start osd.2 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2024-06-11T17:41:17.687+0000 7fe0eccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2024-06-11T17:41:17.743+0000 7fe0eccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2024-06-11T17:41:17.747+0000 7fe0eccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 0 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2024-06-11T17:41:18.299+0000 7fe0eccdd7c0 -1 Falling back to public interface 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.806 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 1 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:22.807 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 3 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:22.808 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2024-06-11T17:41:22.799+0000 7feovery_ops 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:22.935 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2024-06-11T17:41:17.683+0000 7f4b3d9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2024-06-11T17:41:17.743+0000 7f4b3d9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2024-06-11T17:41:17.747+0000 7f4b3d9c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 0 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2024-06-11T17:41:18.291+0000 7f4b3d9c77c0 -1 Falling back to public interface 2024-06-11T17:41:22.936 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 1 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 3 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2024-06-11T17:41:22.931+0000 7f40eccdd7c0 -1 osd.2 75 log_to_monitors true 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 4 2024-06-11T17:41:24.590 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: osd.2 up in weight 1 up_from 81 up_thru 53 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/1360569088,v1:127.0.0.1:6819/1360569088] [v2:127.0.0.1:6820/1360569088,v1:127.0.0.1:6821/1360569088] exists,up b71c9df3-e3f9-4b1d-ab97-edd4f2069cfa 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:24.591 INFO:tasks.workunit.client.0.smithi012.stderr:184381: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:24.592 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 1 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 2 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: 3' 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836545 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836545 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545' 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalonb3d9c77c0 -1 osd.1 75 log_to_monitors true 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:24.593 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 4 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: osd.1 up in weight 1 up_from 81 up_thru 73 down_at 76 last_clean_interval [72,75) [v2:127.0.0.1:6810/2669719221,v1:127.0.0.1:6811/2669719221] [v2:127.0.0.1:6812/2669719221,v1:127.0.0.1:6813/2669719221] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:24.594 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:24.595 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 1 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 2 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: 3' 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836544 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836544 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544' 2024-06-11T17:41:24.596 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:27.488 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:27.488 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350979 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350979 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-347892350979' 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350979 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350979 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-347892350979 2-347892350979' 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906956 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906956 2024-06-11T17:41:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-347892350979 2-347892350979 3-274877906956' 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836544 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836544 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836544 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: waiting osd.0 seq 21474836544 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836543 -lt 21474836544 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:27.490 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836546 -lt 21474836544 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-347892350979 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-347892350979 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:18437e/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350978 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350978 2024-06-11T17:41:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-347892350978' 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350978 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350978 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-347892350978 2-347892350978' 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906955 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906955 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836545 1-347892350978 2-347892350978 3-274877906955' 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:27.492 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836545 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836545 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836545 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: waiting osd.0 seq 21474836545 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836543 -lt 21474836545 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:27.493 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836546 -lt 21474836545 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-347892350978 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-347892350978 2024-06-11T17:41:27.494 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350978 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: waiting osd.1 seq 347892350978 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350979 -lt 347892350978 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350978 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350978 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350978 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: waiting osd.2 seq 347892350978 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350979 -lt 347892350978 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-274877906955 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-274877906955 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:28.913 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906955 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 274877906955' 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: waiting osd.3 seq 274877906955 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906957 -lt 274877906955 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:28.914 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(8: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350979 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 347892350979' 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: waiting osd.1 seq 347892350979 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350979 -lt 347892350979 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350979 2024-06-11T17:41:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350979 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350979 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350979' 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: waiting osd.2 seq 347892350979 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350979 -lt 347892350979 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-274877906956 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-274877906956 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906956 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: waiting osd.3 seq 274877906956 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906957 -lt 274877906956 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:28.961 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:30.398 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:30.398 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:30.398 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:30.398 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-11T17:41:30.398 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:30.399 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:41:30.400 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:41:30.482 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_seccontains("stale") | not)' 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-11T17:41:30.483 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:30.484 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:41:30.485 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:41:31.825 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:31.826 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:41:31.827 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.p 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:41:31.906 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:41:31.907 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:31.908 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph -g_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:41:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:33.563 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s-format json pg dump pgs 2024-06-11T17:41:33.563 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:33.563 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T17:41:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:33.565 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:33.883 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:33.883 INFO:tasks.workunit.client.0.smithi012.stderr:184381: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:33.883 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:33.884 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:33.884 INFO:tasks.workunit.client.0.smithi012.stderr:184381: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:33.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:h:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:184378: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:184378: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:33.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 184377 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:41:33.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:41:34.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:41:34.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-11T17:41:34.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:41:34.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:41:34.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 187226"' 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 187226' 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 187227"' 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 187227' 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:41:34.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 187226 187227' 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 187226 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/187229: /' 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:34.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/187231: /' 2024-06-11T17:41:38.447 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.447 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:38.448 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.449 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:38.450 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:41:38.450 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:38.450 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:41:38.450 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:38.450 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:38.453 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:38.454 INFO:tasks.workunit.client.0.smithi012.stderr:187229: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:38.455 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:41:38.456 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:41:38.456 INFO:tasks.workunit.client.0.smithi012.stderr:187229: start osd.3 2024-06-11T17:41:38.456 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:41:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:38.786 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:38.787 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:41:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:41:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:38.791 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: start osd.1 2024-06-11T17:41:38.792 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:41:43.332 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:41:43.332 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:41:43.332 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2024-06-11T17:41:38.491+0000 7fc160e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2024-06-11T17:41:38.519+0000 7fc160e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2024-06-11T17:41:38.523+0000 7fc160e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.333 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 0 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2024-06-11T17:41:39.047+0000 7fc160e307c0 -1 Falling back to public interface 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 1 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2024-06-11T17:41:42.043+0000 7fc160e307c0 -1 osd.3 84 log_to_monitors true 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 3 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:41:43.533 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2024-06-11T17:41:38.819+0000 7fabe90e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2024-06-11T17:41:38.823+0000 7fabe90e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2024-06-11T17:41:38.823+0000 7fabe90e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:41:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 0 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2024-06-11T17:41:39.375+0000 7fabe90e07c0 -1 Falling back to public interface 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 1 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2024-06-11T17:41:42.863+0000 7fabe90e07c0 -1 osd.1 84 log_to_monitors true 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:43.536 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:41:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 3 2024-06-11T17:41:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:45.737 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 4 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: osd.1 up in weight 1 up_from 90 up_thru 81 down_at 85 last_clean_interval [81,84) [v2:127.0.0.1:6826/2589750413,v1:127.0.0.1:6827/2589750413] [v2:127.0.0.1:6828/2589750413,v1:127.0.0.1:6829/2589750413] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 1 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 2 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: 3' 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836550 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836550 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550' 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:41:45.740 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 4 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: osd.3 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [64,84) [v2:127.0.0.1:6810/1271228977,v1:127.0.0.1:6811/1271228977] [v2:127.0.0.1:6812/1271228977,v1:127.0.0.1:6813/1271228977] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:41:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 1 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 2 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: 3' 2024-06-11T17:41:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836551 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836551 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836551' 2024-06-11T17:41:45.743 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:47.313 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056642 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056642 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-386547056642' 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350985 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350985 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-386547056642 2-347892350985' 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089346 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089346 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-386547056642 2-347892350985 3-382252089346' 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836550 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836550 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836550 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: waiting osd.0 seq 21474836550 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836552 -lt 21474836550 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:47.315 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-386547056642 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-386547056642 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056642 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: waiting osd.1 seq 386547056642 2024-06-11T17:41:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:47.319 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223pers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:41:47.319 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056643 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056643 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836551 1-386547056643' 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350984 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350984 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836551 1-386547056643 2-347892350984' 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089347 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089347 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836551 1-386547056643 2-347892350984 3-382252089347' 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836551 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836551 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836551 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: waiting osd.0 seq 21474836551 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836552 -lt 21474836551 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-386547056643 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:41:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-386547056643 2024-06-11T17:41:47.322 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:47.322 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056643 2024-06-11T17:41:47.322 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 386547056643' 2024-06-11T17:41:47.322 INFO:tasks.workunit.client.0.smithi012.stderr:187229: waiting osd.1 seq 386547056643 2024-06-11T17:41:47.322 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 386547056643 -lt 386547056642 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350985 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350985 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350985 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: waiting osd.2 seq 347892350985 2024-06-11T17:41:49.746 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350985 -lt 347892350985 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-382252089346 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-382252089346 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089346 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 382252089346' 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: waiting osd.3 seq 382252089346 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 382252089346 2024-06-11T17:41:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089348 -lt 382252089346 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:49.831 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") |0: flush_pg_stats: test 386547056643 -lt 386547056643 2024-06-11T17:41:49.831 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350984 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350984 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350984 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350984' 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: waiting osd.2 seq 347892350984 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350985 -lt 347892350984 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-382252089347 2024-06-11T17:41:49.832 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-382252089347 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089347 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 382252089347' 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: waiting osd.3 seq 382252089347 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 382252089347 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089348 -lt 382252089347 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:41:49.833 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:41:49.834 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:41:50.497 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:41:50.497 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:187231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:50.613 INFO:tasks.workunit.client.0.smithi012.stderr: not)' 2024-06-11T17:41:50.613 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:41:50.613 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:187229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 187227 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:41:50.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:41:50.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:41:50.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:41:50.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:41:50.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:41:50.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:41:50.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:41:50.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:50.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:50.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:50.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:51.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:51.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:41:51.559 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:41:51.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:51.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:41:51.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:51.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:41:51.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:41:51.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:41:51.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:41:51.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:51.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:51.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:51.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:51.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:41:15.951702+0000 '>' 2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:51.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:41:52.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:41:52.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:41:52.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:41:52.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:52.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:52.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:52.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:53.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:41:15.951702+0000 '>' 2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:53.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:41:54.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:41:54.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:41:54.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:41:54.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:41:54.239 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:41:54.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:41:54.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:41:54.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:41:52.201945+0000 '>' 2024-06-11T17:41:15.951702+0000 2024-06-11T17:41:54.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 190079"' 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 190079' 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 190080"' 2024-06-11T17:41:54.582 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 190080' 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 190079 190080' 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 190079 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/190082: /' 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:54.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/190085: /' 2024-06-11T17:41:56.475 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.475 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:56.475 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T17:41:56.475 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:56.475 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.476 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: _ 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: hinfo_key 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: snapset 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:41:56.478 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:56.478 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:56.480 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:56.481 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T17:41:56.482 INFO:tasks.workunit.client.0.smithi012.stderr:190085: start osd.1 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:41:56.487 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:41:56.488 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: _ 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: hinfo_key 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: snapset 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:41:56.489 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:41:56.491 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:41:56.491 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:41:56.491 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:41:56.491 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:41:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:41:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:190082: start osd.3 2024-06-11T17:42:00.916 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:42:00.916 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:42:00.916 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:42:00.916 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2024-06-11T17:41:56.531+0000 7f1fb26887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2024-06-11T17:41:56.535+0000 7f1fb26887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2024-06-11T17:41:56.539+0000 7f1fb26887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 0 2024-06-11T17:42:00.917 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2024-06-11T17:41:57.087+0000 7f1fb26887c0 -1 Falling back to public interface 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 1 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:00.918 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 3 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:00.919 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:42:01.370 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2024-06-11T17:42:00.911+0000 7f1fbovery_ops 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2024-06-11T17:41:56.515+0000 7f8d2395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2024-06-11T17:41:56.527+0000 7f8d2395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2024-06-11T17:41:56.527+0000 7f8d2395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:42:01.371 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 0 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2024-06-11T17:41:57.087+0000 7f8d2395d7c0 -1 Falling back to public interface 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:42:01.372 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 1 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:42:01.373 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 3 2024-06-11T17:42:01.374 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:01.374 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:42:01.374 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2024-06-11T17:42:01.151+0000 7f8d2395d7c0 -1 osd.1 92 log_to_monitors true 2024-06-11T17:42:03.953 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c26887c0 -1 osd.3 92 log_to_monitors true 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 4 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:42:03.954 INFO:tasks.workunit.client.0.smithi012.stderr:190082: osd.3 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [89,92) [v2:127.0.0.1:6810/1559007062,v1:127.0.0.1:6812/1559007062] [v2:127.0.0.1:6814/1559007062,v1:127.0.0.1:6816/1559007062] exists,up 1d37955f-ed1a-433e-abb1-df64da9a5f53 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:42:03.955 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 1 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 2 2024-06-11T17:42:03.956 INFO:tasks.workunit.client.0.smithi012.stderr:190082: 3' 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836556 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836556 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836556' 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperseph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:03.957 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 4 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: osd.1 up in weight 1 up_from 97 up_thru 90 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6811/1227342228,v1:127.0.0.1:6813/1227342228] [v2:127.0.0.1:6815/1227342228,v1:127.0.0.1:6817/1227342228] exists,up 41c497b1-27ae-4b67-8d11-012ee9169fff 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:42:03.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:42:03.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 1 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 2 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: 3' 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836557 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836557 2024-06-11T17:42:03.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836557' 2024-06-11T17:42:03.961 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.141 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalon.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=416611827714 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 416611827714 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836556 1-416611827714' 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350991 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350991 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836556 1-416611827714 2-347892350991' 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:42:05.142 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860418 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860418 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836556 1-416611827714 2-347892350991 3-412316860418' 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836556 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836556 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836556 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: waiting osd.0 seq 21474836556 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836557 -lt 21474836556 2024-06-11T17:42:05.143 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-416611827714 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-416611827714 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=416611827714 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 416611827714' 2024-06-11T17:42:05.144 INFO:tasks.workunit.client.0.smithi012.stderr:190082: waiting osd.1 seq 416611827714 2024-06-11T17:42:05.502 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_e/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=416611827715 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 416611827715 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836557 1-416611827715' 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350990 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350990 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836557 1-416611827715 2-347892350990' 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860419 2024-06-11T17:42:05.503 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860419 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836557 1-416611827715 2-347892350990 3-412316860419' 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836557 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836557 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836557 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: waiting osd.0 seq 21474836557 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836557 -lt 21474836557 2024-06-11T17:42:05.504 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-416611827715 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-416611827715 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=416611827715 2024-06-11T17:42:05.505 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2024-06-11T17:42:05.506 INFO:tasks.workunit.client.0.smithi012.stderr:190085: waiting osd.1 seq 416611827715 2024-06-11T17:42:05.506 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 416611827715 -lt 416611827715 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350990 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350990 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:07.958 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350990 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350990' 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: waiting osd.2 seq 347892350990 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350989 -lt 347892350990 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350991 -lt 347892350990 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-412316860419 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-412316860419 2024-06-11T17:42:07.959 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860419 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 412316860419' 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: waiting osd.3 seq 412316860419 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860419 -lt 412316860419 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:42:07.960 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:42:07.972 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='selecstats: ceph osd last-stat-seq 1 2024-06-11T17:42:07.972 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 416611827715 -lt 416611827714 2024-06-11T17:42:07.972 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:07.972 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350991 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350991 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350991 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350991' 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: waiting osd.2 seq 347892350991 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350989 -lt 347892350991 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:42:07.973 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350991 -lt 347892350991 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-412316860418 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-412316860418 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860418 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 412316860418' 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: waiting osd.3 seq 412316860418 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860419 -lt 412316860418 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:42:07.974 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:42:07.975 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:42:07.975 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:42:07.975 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:42:07.975 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:42:07.975 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:42:08.787 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:42:08.787 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:190082: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:42:08.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:42:08.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:42:08.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 190080 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:t(contains("stale") | not)' 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T17:42:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:42:09.544 INFO:tasks.workunit.client.0.smithi012.stderr:190085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:190085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:42:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:42:09.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:42:09.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:42:09.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:42:09.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:42:09.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:42:09.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:42:09.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:42:09.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:42:09.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:42:09.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:42:09.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:42:09.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:42:09.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:42:09.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:42:09.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:42:09.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:42:09.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:42:09.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:42:09.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:42:09.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:42:09.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:42:09.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:42:09.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:42:09.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:42:09.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:42:09.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:09.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:09.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:42:09.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:42:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:42:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:42:09.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:42:09.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:42:09.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:42:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:42:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:42:09.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:42:09.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:42:09.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:42:09.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:42:09.951 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:42:09.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:42:09.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:42:09.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:42:09.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:42:09.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:42:09.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:42:09.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:42:09.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:42:09.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:42:09.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:42:09.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:42:09.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:09.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:09.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:42:09.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:42:09.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:42:09.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:42:09.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:42:09.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:09.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:09.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:42:09.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-11T17:42:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:42:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:42:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:42:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:42:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:42:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:42:10.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:42:10.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:42:10.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:42:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:42:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:10.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:10.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:42:10.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:42:10.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:42:10.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:42:10.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:42:10.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:42:10.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:42:10.705 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:42:10.705 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:42:10.705 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:42:10.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:42:10.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:42:10.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:42:10.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:42:10.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:42:10.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:42:10.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:42:10.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-11T17:42:10.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:42:10.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:42:10.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:42:10.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:42:10.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:42:10.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:42:11.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:42:11.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:42:11.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:42:11.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:42:11.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:11.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:11.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:11.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:42:11.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:42:11.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-11T17:42:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:42:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-11T17:42:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:42:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:42:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:42:11.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:42:11.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:42:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:42:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:11.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:42:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=18a6d409-025d-407c-ab36-d1f2afa3bae0 2024-06-11T17:42:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 18a6d409-025d-407c-ab36-d1f2afa3bae0' 2024-06-11T17:42:11.934 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 18a6d409-025d-407c-ab36-d1f2afa3bae0 2024-06-11T17:42:11.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:42:11.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzjGhmfqqwOBAAFaNCGvESHsFEYUyL7aWYnA== 2024-06-11T17:42:11.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzjGhmfqqwOBAAFaNCGvESHsFEYUyL7aWYnA=="}' 2024-06-11T17:42:11.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 18a6d409-025d-407c-ab36-d1f2afa3bae0 -i td/osd-scrub-repair/0/new.json 2024-06-11T17:42:12.238 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:42:12.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:42:12.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzjGhmfqqwOBAAFaNCGvESHsFEYUyL7aWYnA== --osd-uuid 18a6d409-025d-407c-ab36-d1f2afa3bae0 2024-06-11T17:42:12.286 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.283+0000 7f455ed267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:12.322 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.319+0000 7f455ed267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:12.324 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.319+0000 7f455ed267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:12.325 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.319+0000 7f455ed267c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:42:12.325 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.319+0000 7f455ed267c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:42:12.325 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.319+0000 7f455ed267c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:42:12.439 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:12.435+0000 7f455ed267c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:42:19.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:42:19.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:42:19.954 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:42:19.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:42:19.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:42:20.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:42:20.360 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:42:20.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:42:20.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:42:20.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:42:20.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:42:20.392 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:20.387+0000 7f0539aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:20.437 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:20.431+0000 7f0539aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:20.438 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:20.435+0000 7f0539aec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:20.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:42:20.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:42:20.684 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:20.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:21.007 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:21.003+0000 7f0539aec7c0 -1 Falling back to public interface 2024-06-11T17:42:21.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:22.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:22.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:22.009 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:42:22.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:42:22.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:22.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:22.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:23.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:23.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:23.346 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:42:23.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:42:23.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:23.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:23.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:23.815 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:23.811+0000 7f0539aec7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:42:24.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:24.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:24.671 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:42:24.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:42:24.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:24.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:25.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:26.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:26.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:26.054 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:42:26.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:42:26.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:26.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:26.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:27.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:27.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:27.391 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:42:27.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:42:27.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:27.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:27.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:28.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:28.886 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:42:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:42:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:29.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:30.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:30.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:30.227 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:42:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:42:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:30.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:31.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:31.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:31.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:42:31.563 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:42:31.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:31.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:31.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:32.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:32.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:32.989 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:42:32.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:42:32.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:32.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:33.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:34.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:34.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:34.328 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:42:34.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:42:34.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:34.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:34.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:35.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:35.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:35.745 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:42:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:42:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:36.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:37.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:37.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:37.205 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:42:37.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:42:37.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:37.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:37.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:38.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:38.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:38.667 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:42:38.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:42:38.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:38.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:39.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:40.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:40.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:40.185 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:42:40.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:42:40.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:40.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:40.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:41.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:41.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:41.524 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:42:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:42:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:42.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:43.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:43.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:43.088 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:42:43.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:42:43.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:43.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:43.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:44.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:44.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:44.992 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:42:44.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:42:44.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:44.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:45.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:46.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:46.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:46.415 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:42:46.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:42:46.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:46.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:47.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:48.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:48.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:48.032 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:42:48.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:42:48.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:48.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:48.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:49.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:49.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:49.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:42:49.663 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:42:49.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:49.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:50.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:51.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:51.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:51.388 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:42:51.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:42:51.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:51.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:52.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:53.320 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:42:53.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:42:53.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:53.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:53.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:42:54.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:42:54.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:42:54.807 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:42:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:42:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:42:54.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:42:55.262 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2010196391,v1:127.0.0.1:6803/2010196391] [v2:127.0.0.1:6804/2010196391,v1:127.0.0.1:6805/2010196391] exists,up 18a6d409-025d-407c-ab36-d1f2afa3bae0 2024-06-11T17:42:55.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:42:55.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:42:55.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:42:55.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:42:55.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:42:55.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:42:55.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:42:55.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:42:55.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:42:55.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:42:55.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:42:55.271 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:42:55.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:42:55.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:42:55.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:42:55.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:42:55.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:42:55.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:42:55.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:42:55.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fba279b-fd0c-4a4c-b297-d3e598f12792 2024-06-11T17:42:55.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7fba279b-fd0c-4a4c-b297-d3e598f12792' 2024-06-11T17:42:55.276 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 7fba279b-fd0c-4a4c-b297-d3e598f12792 2024-06-11T17:42:55.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:42:55.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfjGhm9mAZERAAb8qrs2DBawl63IILj2B5Dw== 2024-06-11T17:42:55.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfjGhm9mAZERAAb8qrs2DBawl63IILj2B5Dw=="}' 2024-06-11T17:42:55.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fba279b-fd0c-4a4c-b297-d3e598f12792 -i td/osd-scrub-repair/1/new.json 2024-06-11T17:42:55.704 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:42:55.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:42:55.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfjGhm9mAZERAAb8qrs2DBawl63IILj2B5Dw== --osd-uuid 7fba279b-fd0c-4a4c-b297-d3e598f12792 2024-06-11T17:42:55.763 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.762+0000 7fc8e20da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:55.847 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.846+0000 7fc8e20da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:55.849 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.850+0000 7fc8e20da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:42:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.850+0000 7fc8e20da7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:42:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.850+0000 7fc8e20da7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:42:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.850+0000 7fc8e20da7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:42:55.925 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:42:55.922+0000 7fc8e20da7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:43:03.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:43:03.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:43:03.985 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:43:03.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:43:03.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:43:04.396 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:43:04.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:43:04.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:43:04.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:43:04.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:43:04.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:43:04.429 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:04.426+0000 7fd94eb267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:04.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:04.474+0000 7fd94eb267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:04.478 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:04.478+0000 7fd94eb267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:43:04.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:04.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:43:04.714 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:43:04.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:04.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:05.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:05.064 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:05.062+0000 7fd94eb267c0 -1 Falling back to public interface 2024-06-11T17:43:06.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:06.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:06.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:43:06.019 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:43:06.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:06.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:06.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:07.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:07.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:07.347 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:43:07.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:43:07.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:07.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:07.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:07.981 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:07.978+0000 7fd94eb267c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:43:08.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:08.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:08.680 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:43:08.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:43:08.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:08.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:09.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:10.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:10.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:10.156 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:43:10.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:43:10.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:10.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:10.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:11.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:11.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:11.499 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:43:11.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:43:11.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:11.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:11.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:12.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:12.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:12.911 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:43:12.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:43:12.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:12.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:13.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:14.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:14.471 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:43:14.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:43:14.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:14.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:14.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:15.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:15.844 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:43:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:43:15.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:15.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:16.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:17.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:17.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:17.368 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:43:17.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:43:17.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:17.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:17.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:18.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:18.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:18.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:43:18.896 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:43:18.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:18.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:19.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:20.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:20.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:43:20.500 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:43:20.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:20.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:20.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:21.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:21.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:21.882 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:43:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:43:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:22.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:23.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:23.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:23.289 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:43:23.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:43:23.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:23.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:23.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:24.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:24.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:24.677 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:43:24.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:43:24.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:24.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:25.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:26.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:26.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:26.218 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:43:26.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:43:26.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:26.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:26.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:27.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:27.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:27.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:43:27.551 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:43:27.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:27.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:27.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:28.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:28.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:28.948 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:43:28.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:43:28.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:28.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:29.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:30.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:30.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:30.434 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:43:30.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:43:30.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:30.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:31.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:32.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:32.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:32.048 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:43:32.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:43:32.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:32.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:32.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:33.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:33.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:33.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:43:33.670 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:43:33.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:33.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:34.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:35.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:35.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:35.170 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:43:35.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:43:35.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:35.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:36.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:37.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:37.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:37.141 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:43:37.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:43:37.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:37.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:38.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:39.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:39.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:39.194 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:43:39.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:43:39.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:39.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:43:39.533 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3546470015,v1:127.0.0.1:6811/3546470015] [v2:127.0.0.1:6812/3546470015,v1:127.0.0.1:6813/3546470015] exists,up 7fba279b-fd0c-4a4c-b297-d3e598f12792 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:43:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:43:39.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:43:39.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:43:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:43:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:43:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:43:39.538 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:43:39.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:43:39.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:43:39.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:43:39.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:43:39.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84b01a65-4762-4abd-bfd6-fb0f5ea626b9 2024-06-11T17:43:39.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 84b01a65-4762-4abd-bfd6-fb0f5ea626b9' 2024-06-11T17:43:39.540 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 84b01a65-4762-4abd-bfd6-fb0f5ea626b9 2024-06-11T17:43:39.541 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:43:39.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLjGhmqEaEIRAArRcoKX96dl8fVZBO7LZ4Nw== 2024-06-11T17:43:39.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLjGhmqEaEIRAArRcoKX96dl8fVZBO7LZ4Nw=="}' 2024-06-11T17:43:39.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84b01a65-4762-4abd-bfd6-fb0f5ea626b9 -i td/osd-scrub-repair/2/new.json 2024-06-11T17:43:39.956 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:43:39.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:43:39.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLjGhmqEaEIRAArRcoKX96dl8fVZBO7LZ4Nw== --osd-uuid 84b01a65-4762-4abd-bfd6-fb0f5ea626b9 2024-06-11T17:43:40.013 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.010+0000 7f5a3c0c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:40.083 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.082+0000 7f5a3c0c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:40.084 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.082+0000 7f5a3c0c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:40.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.082+0000 7f5a3c0c77c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:43:40.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.082+0000 7f5a3c0c77c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:43:40.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.082+0000 7f5a3c0c77c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:43:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:40.174+0000 7f5a3c0c77c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:43:48.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:43:48.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:43:48.337 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:43:48.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:43:48.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:43:48.748 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:43:48.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:43:48.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:43:48.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:43:48.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:43:48.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:43:48.782 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:48.782+0000 7f787af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:48.813 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:48.810+0000 7f787af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:48.815 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:48.814+0000 7f787af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:43:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:43:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:43:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:43:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:43:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:43:49.165 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:43:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:43:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:49.371 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:49.370+0000 7f787af9e7c0 -1 Falling back to public interface 2024-06-11T17:43:49.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:50.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:50.498 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:43:50.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:43:50.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:50.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:50.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:51.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:51.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:51.835 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:43:51.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:43:51.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:51.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:52.157 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:43:52.158+0000 7f787af9e7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:43:52.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:53.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:53.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:53.174 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:43:53.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:43:53.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:53.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:53.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:54.660 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:43:54.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:43:54.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:54.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:54.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:55.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:55.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:55.994 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:43:55.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:43:55.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:55.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:56.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:57.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:57.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:57.402 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:43:57.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:43:57.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:43:58.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:43:58.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:43:58.855 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:43:58.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:43:58.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:43:58.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:43:59.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:00.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:00.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:00.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:44:00.468 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:44:00.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:00.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:00.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:01.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:01.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:01.809 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:44:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:44:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:01.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:02.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:03.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:03.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:03.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:44:03.326 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:44:03.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:03.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:03.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:04.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:04.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:04.855 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:44:04.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:44:04.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:04.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:05.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:06.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:06.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:06.310 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:44:06.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:44:06.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:06.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:06.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:07.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:07.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:07.647 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:44:07.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:44:07.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:07.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:08.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:09.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:09.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:09.034 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:44:09.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:44:09.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:09.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:09.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:10.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:10.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:10.520 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:44:10.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:44:10.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:10.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:10.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:11.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:11.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:11.849 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:44:11.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:44:11.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:11.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:12.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:13.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:13.304 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:44:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:44:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:14.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:15.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:15.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:15.053 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:44:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:44:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:15.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:16.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:17.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:17.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:17.011 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:44:17.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:44:17.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:17.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:17.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:18.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:18.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:18.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:44:18.981 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:44:18.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:18.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:19.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:20.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:20.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:20.365 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:44:20.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:44:20.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:20.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:21.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:22.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:22.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:22.108 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:44:22.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:44:22.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:22.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:22.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:23.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:23.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:23.569 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:44:23.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:44:23.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:23.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:44:23.940 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1423252796,v1:127.0.0.1:6819/1423252796] [v2:127.0.0.1:6820/1423252796,v1:127.0.0.1:6821/1423252796] exists,up 84b01a65-4762-4abd-bfd6-fb0f5ea626b9 2024-06-11T17:44:23.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:44:23.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:44:23.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:44:23.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:44:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:44:23.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:44:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:44:23.947 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:44:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8c4184ca-c97f-451d-9abf-0e957cb760ea' 2024-06-11T17:44:23.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:44:23.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3jGhmKT7EORAAuf5pFcQZzzGBzigODSXk6Q== 2024-06-11T17:44:23.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3jGhmKT7EORAAuf5pFcQZzzGBzigODSXk6Q=="}' 2024-06-11T17:44:23.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c4184ca-c97f-451d-9abf-0e957cb760ea -i td/osd-scrub-repair/3/new.json 2024-06-11T17:44:24.385 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:44:24.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-11T17:44:24.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3jGhmKT7EORAAuf5pFcQZzzGBzigODSXk6Q== --osd-uuid 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:44:24.445 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.442+0000 7f9bfe4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:24.520 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.518+0000 7f9bfe4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:24.522 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.522+0000 7f9bfe4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:24.522 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.522+0000 7f9bfe4747c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:44:24.522 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.522+0000 7f9bfe4747c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:44:24.522 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.522+0000 7f9bfe4747c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:44:24.604 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:24.602+0000 7f9bfe4747c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-11T17:44:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-11T17:44:33.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:44:33.156 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:44:33.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:44:33.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:44:33.576 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:44:33.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:44:33.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:44:33.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:44:33.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:44:33.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:44:33.611 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:33.610+0000 7fa636c2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:33.637 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:33.634+0000 7fa636c2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:33.638 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:33.638+0000 7fa636c2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:44:33.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:33.930 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:44:33.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:44:33.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:33.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:34.187 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:34.186+0000 7fa636c2b7c0 -1 Falling back to public interface 2024-06-11T17:44:34.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:44:35.310 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:44:35.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:35.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:35.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:36.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:36.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:36.650 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:44:36.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:44:36.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:36.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:36.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:37.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:44:37.030+0000 7fa636c2b7c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:44:37.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:37.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:37.994 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:44:37.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:44:37.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:37.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:38.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:39.536 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:44:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:44:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:39.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:39.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:40.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:40.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:40.915 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:44:40.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:44:40.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:40.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:41.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:42.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:42.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:42.557 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:44:42.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:44:42.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:42.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:42.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:43.979 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:44:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:44:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:44.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:45.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:45.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:45.429 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:44:45.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:44:45.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:45.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:45.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:46.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:46.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:46.837 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:44:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:44:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:47.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:48.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:48.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:48.383 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:44:48.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:44:48.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:48.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:48.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:49.717 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:44:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:44:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:50.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:51.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:51.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:51.209 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:44:51.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:44:51.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:51.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:51.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:52.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:52.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:52.551 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:44:52.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:44:52.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:52.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:53.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:53.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:53.941 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:44:53.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:44:53.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:53.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:54.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:55.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:55.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:55.368 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:44:55.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:44:55.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:55.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:56.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:57.479 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:44:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:44:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:57.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:44:58.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:44:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:44:58.912 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:44:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:44:58.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:44:58.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:44:59.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:00.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:00.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:00.489 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:45:00.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:45:00.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:00.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:01.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:02.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:02.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:02.090 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:45:02.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:45:02.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:02.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:02.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:03.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:03.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:03.726 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:45:03.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:45:03.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:03.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:04.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:05.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:05.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:05.091 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:45:05.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:45:05.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:05.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:05.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:06.716 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:45:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:45:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:06.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:07.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:08.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:08.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:08.328 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:45:08.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:45:08.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:08.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:45:08.692 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/823366817,v1:127.0.0.1:6827/823366817] [v2:127.0.0.1:6828/823366817,v1:127.0.0.1:6829/823366817] exists,up 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:45:08.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:45:08.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:45:08.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:45:08.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:45:08.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-11T17:45:08.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-11T17:45:08.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:45:08.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:45:08.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:45:08.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:45:08.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:45:08.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:45:08.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:45:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:45:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:45:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:45:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:45:08.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:45:08.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-11T17:45:08.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:45:08.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a02be0a2-2584-4c46-bd8f-d1ea2148523e 2024-06-11T17:45:08.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a02be0a2-2584-4c46-bd8f-d1ea2148523e' 2024-06-11T17:45:08.708 INFO:tasks.workunit.client.0.smithi012.stdout:add osd4 a02be0a2-2584-4c46-bd8f-d1ea2148523e 2024-06-11T17:45:08.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:45:08.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkjWhmy1LyKhAAyNjIOqvozknENb/OEUAnJA== 2024-06-11T17:45:08.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkjWhmy1LyKhAAyNjIOqvozknENb/OEUAnJA=="}' 2024-06-11T17:45:08.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a02be0a2-2584-4c46-bd8f-d1ea2148523e -i td/osd-scrub-repair/4/new.json 2024-06-11T17:45:09.344 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:45:09.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-11T17:45:09.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkjWhmy1LyKhAAyNjIOqvozknENb/OEUAnJA== --osd-uuid a02be0a2-2584-4c46-bd8f-d1ea2148523e 2024-06-11T17:45:09.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.405+0000 7f32a18e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:09.466 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.465+0000 7f32a18e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:09.468 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.465+0000 7f32a18e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:09.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.465+0000 7f32a18e27c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:45:09.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.465+0000 7f32a18e27c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:45:09.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.465+0000 7f32a18e27c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:45:09.545 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:09.541+0000 7f32a18e27c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-11T17:45:17.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-11T17:45:17.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:45:17.402 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd4 key to auth repository 2024-06-11T17:45:17.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-11T17:45:17.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:45:17.820 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.4 2024-06-11T17:45:17.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-11T17:45:17.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:45:17.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:45:17.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:45:17.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:45:17.854 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:17.853+0000 7f15caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:17.887 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:17.885+0000 7f15caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:17.888 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:17.885+0000 7f15caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:45:18.186 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:45:18.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:18.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:18.440 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:18.437+0000 7f15caa157c0 -1 Falling back to public interface 2024-06-11T17:45:18.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:19.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:19.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:19.545 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:45:19.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:45:19.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:19.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:19.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:20.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:20.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:20.876 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:45:20.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:45:20.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:20.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:21.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:21.356 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:21.353+0000 7f15caa157c0 -1 osd.4 0 log_to_monitors true 2024-06-11T17:45:22.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:22.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:22.217 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:45:22.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:45:22.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:22.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:22.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:23.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:23.665 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:45:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:45:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:24.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:25.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:25.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:25.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:45:25.007 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:45:25.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:25.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:25.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:26.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:26.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:26.456 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:45:26.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:45:26.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:26.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:26.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:27.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:27.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:27.790 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:45:27.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:45:27.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:27.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:28.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:29.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:29.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:29.187 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:45:29.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:45:29.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:29.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:29.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:30.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:30.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:30.616 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:45:30.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:45:30.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:30.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:31.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:32.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:32.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:32.057 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:45:32.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:45:32.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:32.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:32.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:33.487 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:45:33.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:45:33.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:33.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:33.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:34.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:34.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:34.826 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:45:34.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:45:34.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:34.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:35.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:36.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:36.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:36.238 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:45:36.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:45:36.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:36.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:36.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:37.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:45:37.645 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:45:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:37.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:37.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:38.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:38.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:38.981 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:45:38.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:45:38.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:38.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:40.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:40.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:40.617 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:45:40.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:45:40.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:40.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:41.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:42.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:42.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:42.275 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:45:42.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:45:42.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:42.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:43.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:44.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:44.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:44.061 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:45:44.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:45:44.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:44.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:44.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:45.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:45.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:45.539 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:45:45.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:45:45.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:45.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:46.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:47.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:47.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:47.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:45:47.202 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:45:47.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:47.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:47.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:48.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:48.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:48.956 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:45:48.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:45:48.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:48.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:49.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:50.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:50.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:50.914 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:45:50.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:45:50.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:50.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:51.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:45:52.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:45:52.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:45:52.444 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:45:52.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:45:52.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:45:52.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:45:52.840 INFO:tasks.workunit.client.0.smithi012.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4248499851,v1:127.0.0.1:6835/4248499851] [v2:127.0.0.1:6836/4248499851,v1:127.0.0.1:6837/4248499851] exists,up a02be0a2-2584-4c46-bd8f-d1ea2148523e 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:45:52.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:45:52.844 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:45:52.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:45:52.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:45:52.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:45:52.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:45:52.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:45:52.848 INFO:tasks.workunit.client.0.smithi012.stdout:add osd5 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:45:52.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb' 2024-06-11T17:45:52.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:45:52.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQjWhmuHS0MxAA4RVvvmLno+GIEuD4ONyWcQ== 2024-06-11T17:45:52.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQjWhmuHS0MxAA4RVvvmLno+GIEuD4ONyWcQ=="}' 2024-06-11T17:45:52.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb -i td/osd-scrub-repair/5/new.json 2024-06-11T17:45:53.361 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:45:53.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-11T17:45:53.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQjWhmuHS0MxAA4RVvvmLno+GIEuD4ONyWcQ== --osd-uuid 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:45:53.421 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.421+0000 7f34adcbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:53.513 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.509+0000 7f34adcbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:53.515 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.513+0000 7f34adcbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:45:53.516 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.513+0000 7f34adcbf7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:45:53.516 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.513+0000 7f34adcbf7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:45:53.516 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.513+0000 7f34adcbf7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:45:53.600 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:45:53.597+0000 7f34adcbf7c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-11T17:46:01.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-11T17:46:01.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:46:01.985 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd5 key to auth repository 2024-06-11T17:46:01.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-11T17:46:01.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:46:02.406 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.5 2024-06-11T17:46:02.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-11T17:46:02.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:46:02.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:46:02.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:46:02.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:46:02.440 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:02.437+0000 7fc79cddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:02.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:02.473+0000 7fc79cddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:02.473+0000 7fc79cddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:46:02.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:02.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:46:02.759 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:46:02.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:02.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:03.047 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:03.045+0000 7fc79cddf7c0 -1 Falling back to public interface 2024-06-11T17:46:03.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:04.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:04.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:04.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:46:04.153 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:46:04.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:04.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:04.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:05.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:05.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:05.492 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:46:05.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:46:05.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:05.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:05.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:05.921+0000 7fc79cddf7c0 -1 osd.5 0 log_to_monitors true 2024-06-11T17:46:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:46:06.828 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:46:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:08.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:08.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:08.229 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:46:08.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:46:08.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:08.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:08.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:09.760 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:46:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:46:09.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:09.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:10.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:11.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:11.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:11.180 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:46:11.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:46:11.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:11.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:11.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:12.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:12.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:12.519 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:46:12.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:46:12.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:12.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:12.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:13.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:13.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:13.922 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:46:13.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:46:13.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:13.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:14.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:15.385 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:46:15.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:46:15.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:15.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:15.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:16.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:16.868 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:46:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:46:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:16.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:17.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:18.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:18.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:18.540 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:46:18.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:46:18.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:18.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:18.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:19.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:19.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:19.887 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:46:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:46:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:20.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:21.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:21.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:21.314 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:46:21.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:46:21.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:21.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:21.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:22.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:22.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:22.688 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:46:22.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:46:22.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:22.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:23.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:24.271 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:46:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:46:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:25.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:26.027 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:46:26.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:46:26.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:26.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:26.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:27.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:27.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:27.516 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:46:27.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:46:27.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:27.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:28.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:29.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:29.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:29.649 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:46:29.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:46:29.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:29.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:30.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:31.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:31.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:31.447 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:46:31.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:46:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:31.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:32.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:33.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:33.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:33.165 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:46:33.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:46:33.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:33.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:33.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:34.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:34.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:34.782 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:46:34.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:46:34.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:34.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:35.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:36.839 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:46:36.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:46:36.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:36.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:37.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:38.368 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:46:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:46:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:46:38.702 INFO:tasks.workunit.client.0.smithi012.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/973505656,v1:127.0.0.1:6843/973505656] [v2:127.0.0.1:6844/973505656,v1:127.0.0.1:6845/973505656] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:46:38.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:46:38.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:46:38.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-11T17:46:38.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:46:38.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-11T17:46:38.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-11T17:46:38.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:46:38.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:46:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6a0957e-a800-4123-b7d0-624598f109f1 2024-06-11T17:46:38.714 INFO:tasks.workunit.client.0.smithi012.stdout:add osd6 d6a0957e-a800-4123-b7d0-624598f109f1 2024-06-11T17:46:38.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 d6a0957e-a800-4123-b7d0-624598f109f1' 2024-06-11T17:46:38.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:46:38.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+jWhmtdGMKxAAkfZofPLN3346H1UwqpaR5w== 2024-06-11T17:46:38.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+jWhmtdGMKxAAkfZofPLN3346H1UwqpaR5w=="}' 2024-06-11T17:46:38.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6a0957e-a800-4123-b7d0-624598f109f1 -i td/osd-scrub-repair/6/new.json 2024-06-11T17:46:39.141 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:46:39.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-11T17:46:39.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+jWhmtdGMKxAAkfZofPLN3346H1UwqpaR5w== --osd-uuid d6a0957e-a800-4123-b7d0-624598f109f1 2024-06-11T17:46:39.202 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.201+0000 7f5241a607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.277+0000 7f5241a607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.277+0000 7f5241a607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.281+0000 7f5241a607c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:46:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.281+0000 7f5241a607c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:46:39.282 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.281+0000 7f5241a607c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:46:39.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:39.381+0000 7f5241a607c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-11T17:46:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-11T17:46:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:46:46.457 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd6 key to auth repository 2024-06-11T17:46:46.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-11T17:46:46.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:46:46.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-11T17:46:46.867 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.6 2024-06-11T17:46:46.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:46:46.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:46:46.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:46:46.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:46:46.902 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:46.897+0000 7f43a08c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:46.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:46.941+0000 7f43a08c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:46.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:46.941+0000 7f43a08c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:46:47.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-11T17:46:47.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:46:47.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:47.480 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:47.477+0000 7f43a08c67c0 -1 Falling back to public interface 2024-06-11T17:46:47.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:48.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:48.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:48.522 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:46:48.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:46:48.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:48.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:48.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:49.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:49.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:49.860 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:46:49.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:46:49.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:49.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:50.130 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:46:50.128+0000 7f43a08c67c0 -1 osd.6 0 log_to_monitors true 2024-06-11T17:46:50.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:51.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:51.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:51.193 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:46:51.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:46:51.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:51.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:51.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:52.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:52.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:52.747 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:46:52.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:46:52.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:52.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:53.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:54.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:54.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:54.217 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:46:54.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:46:54.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:54.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:54.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:55.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:55.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:55.686 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:46:55.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:46:55.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:55.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:56.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:57.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:57.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:57.179 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:46:57.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:46:57.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:57.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:58.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:58.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:58.516 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:46:58.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:46:58.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:58.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:46:58.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:46:59.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:46:59.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:46:59.859 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:46:59.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:46:59.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:46:59.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:00.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:01.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:01.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:01.261 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:47:01.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:47:01.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:01.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:01.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:02.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:02.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:02.703 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:47:02.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:47:02.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:02.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:03.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:04.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:04.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:04.039 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:47:04.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:47:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:04.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:05.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:05.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:05.393 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:47:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:47:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:06.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:06.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:06.733 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:47:06.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:47:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:07.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:08.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:08.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:08.272 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:47:08.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:47:08.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:08.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:09.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:10.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:10.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:10.029 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:47:10.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:47:10.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:10.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:10.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:11.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:11.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:11.708 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:47:11.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:47:11.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:12.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:13.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:13.321 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:47:13.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:47:13.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:13.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:14.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:15.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:15.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:15.375 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:47:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:47:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:16.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:17.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:17.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:17.068 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:47:17.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:47:17.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:17.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:17.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:18.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:18.865 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:47:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:47:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:18.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:19.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:20.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:20.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:20.224 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:47:20.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:47:20.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:20.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:47:20.560 INFO:tasks.workunit.client.0.smithi012.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1272969762,v1:127.0.0.1:6851/1272969762] [v2:127.0.0.1:6852/1272969762,v1:127.0.0.1:6853/1272969762] exists,up d6a0957e-a800-4123-b7d0-624598f109f1 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-11T17:47:20.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:47:20.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:47:20.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-11T17:47:20.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:47:20.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3508ae0f-e2a7-4ffb-9737-495a76947013 2024-06-11T17:47:20.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 3508ae0f-e2a7-4ffb-9737-495a76947013' 2024-06-11T17:47:20.568 INFO:tasks.workunit.client.0.smithi012.stdout:add osd7 3508ae0f-e2a7-4ffb-9737-495a76947013 2024-06-11T17:47:20.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:47:20.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCojWhm+DvKIhAAnyfGRv8bs+UHMDDfPps+TQ== 2024-06-11T17:47:20.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCojWhm+DvKIhAAnyfGRv8bs+UHMDDfPps+TQ=="}' 2024-06-11T17:47:20.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3508ae0f-e2a7-4ffb-9737-495a76947013 -i td/osd-scrub-repair/7/new.json 2024-06-11T17:47:21.116 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:47:21.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-11T17:47:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCojWhm+DvKIhAAnyfGRv8bs+UHMDDfPps+TQ== --osd-uuid 3508ae0f-e2a7-4ffb-9737-495a76947013 2024-06-11T17:47:21.179 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.176+0000 7fef5bd637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.248+0000 7fef5bd637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:21.253 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.252+0000 7fef5bd637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:21.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.252+0000 7fef5bd637c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:47:21.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.252+0000 7fef5bd637c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:47:21.254 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.252+0000 7fef5bd637c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:47:21.340 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:21.336+0000 7fef5bd637c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-11T17:47:28.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-11T17:47:28.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:47:28.097 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd7 key to auth repository 2024-06-11T17:47:28.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-11T17:47:28.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:47:28.505 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.7 2024-06-11T17:47:28.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-11T17:47:28.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:47:28.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:47:28.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:47:28.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:47:28.538 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:28.536+0000 7f39eee3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:28.580 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:28.576+0000 7f39eee3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:28.582 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:28.580+0000 7f39eee3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-11T17:47:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:47:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-11T17:47:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:47:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:47:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:28.807 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:47:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:47:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:28.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:29.132 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:29.128+0000 7f39eee3c7c0 -1 Falling back to public interface 2024-06-11T17:47:29.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:30.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:30.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:30.218 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:47:30.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:47:30.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:30.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:30.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:31.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:31.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:31.553 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:47:31.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:47:31.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:31.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:31.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:31.652+0000 7f39eee3c7c0 -1 osd.7 0 log_to_monitors true 2024-06-11T17:47:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:32.908 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:47:32.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:47:32.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:32.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:33.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:34.338 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:47:34.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:47:34.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:34.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:34.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:35.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:35.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:35.713 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:47:35.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:47:35.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:35.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:36.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:37.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:37.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:37.145 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:47:37.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:47:37.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:37.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:37.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:38.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:38.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:38.685 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:47:38.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:47:38.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:38.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:39.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:40.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:40.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:40.027 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:47:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:47:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:40.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:41.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:41.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:41.367 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:47:41.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:47:41.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:41.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:41.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:42.735 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:47:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:47:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:44.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:44.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:44.072 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:47:44.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:47:44.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:44.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:44.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:45.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:45.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:45.413 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:47:45.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:47:45.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:45.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:45.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:46.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:46.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:46.910 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:47:46.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:47:46.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:46.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:47.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:48.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:48.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:48.414 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:47:48.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:47:48.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:48.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:50.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:50.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:50.047 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:47:50.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:47:50.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:50.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:50.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:51.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:51.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:51.550 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:47:51.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:47:51.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:51.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:52.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:53.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:53.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:53.286 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:47:53.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:47:53.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:53.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:53.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:54.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:54.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:54.759 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:47:54.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:47:54.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:54.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:55.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:56.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:56.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:56.261 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:47:56.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:47:56.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:56.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:56.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:47:57.848 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:47:58.182 INFO:tasks.workunit.client.0.smithi012.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3055721942,v1:127.0.0.1:6859/3055721942] [v2:127.0.0.1:6860/3055721942,v1:127.0.0.1:6861/3055721942] exists,up 3508ae0f-e2a7-4ffb-9737-495a76947013 2024-06-11T17:47:58.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:47:58.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:47:58.189 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:47:58.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72dace4c-9576-4eb8-ab89-9e7c51378f13 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 72dace4c-9576-4eb8-ab89-9e7c51378f13' 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stdout:add osd8 72dace4c-9576-4eb8-ab89-9e7c51378f13 2024-06-11T17:47:58.191 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:47:58.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOjWhmOfObDBAAJqdhKLwPt+eBZnIYeR1tMg== 2024-06-11T17:47:58.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOjWhmOfObDBAAJqdhKLwPt+eBZnIYeR1tMg=="}' 2024-06-11T17:47:58.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72dace4c-9576-4eb8-ab89-9e7c51378f13 -i td/osd-scrub-repair/8/new.json 2024-06-11T17:47:58.621 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:47:58.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-11T17:47:58.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOjWhmOfObDBAAJqdhKLwPt+eBZnIYeR1tMg== --osd-uuid 72dace4c-9576-4eb8-ab89-9e7c51378f13 2024-06-11T17:47:58.682 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.680+0000 7fb401c117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:58.757 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.752+0000 7fb401c117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:58.759 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.756+0000 7fb401c117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:47:58.760 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.756+0000 7fb401c117c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:47:58.760 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.756+0000 7fb401c117c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:47:58.760 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.756+0000 7fb401c117c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:47:58.842 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:47:58.840+0000 7fb401c117c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-11T17:48:07.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-11T17:48:07.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:48:07.178 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd8 key to auth repository 2024-06-11T17:48:07.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-11T17:48:07.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:48:07.603 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.8 2024-06-11T17:48:07.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-11T17:48:07.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:48:07.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:48:07.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:48:07.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:48:07.638 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:07.636+0000 7f4b410447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:07.678 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:07.676+0000 7f4b410447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:07.679 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:07.676+0000 7f4b410447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:48:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:07.986 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:48:07.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:48:07.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:07.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:08.236 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:08.232+0000 7f4b410447c0 -1 Falling back to public interface 2024-06-11T17:48:08.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:09.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:09.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:09.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:48:09.314 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:48:09.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:09.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:09.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:10.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:10.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:10.651 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:48:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:48:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:10.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:11.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:11.191 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:11.168+0000 7f4b410447c0 -1 osd.8 0 log_to_monitors true 2024-06-11T17:48:12.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:12.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:12.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:48:12.067 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:48:12.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:12.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:12.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:13.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:13.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:13.565 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:48:13.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:48:13.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:13.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:14.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:15.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:15.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:15.015 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:48:15.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:48:15.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:15.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:16.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:16.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:16.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:48:16.344 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:48:16.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:16.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:16.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:17.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:17.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:17.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:48:17.673 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:48:17.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:17.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:18.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:19.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:19.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:19.006 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:48:19.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:48:19.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:19.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:20.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:20.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:20.350 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:48:20.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:48:20.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:20.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:20.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:21.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:21.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:21.908 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:48:21.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:48:21.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:21.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:22.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:23.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:23.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:23.417 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:48:23.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:48:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:23.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:23.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:24.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:24.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:24.857 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:48:24.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:48:24.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:24.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:25.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:26.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:26.325 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:48:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:48:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:26.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:26.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:27.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:27.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:27.662 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:48:27.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:48:27.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:27.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:28.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:29.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:29.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:29.002 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:48:29.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:48:29.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:29.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:29.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:30.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:30.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:30.340 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:48:30.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:48:30.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:30.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:30.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:31.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:31.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:31.953 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:48:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:48:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:31.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:33.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:34.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:34.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:34.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:48:34.171 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:48:34.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:34.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:34.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:35.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:35.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:35.916 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:48:35.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:48:35.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:35.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:36.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:37.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:37.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:37.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:48:37.681 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:48:37.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:37.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:38.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:39.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:39.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:39.118 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:48:39.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:48:39.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:39.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:39.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:40.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:40.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:40.503 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:48:40.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:48:40.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:40.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:40.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:41.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:41.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:41.863 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:48:41.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:48:41.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:41.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:42.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:43.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:43.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:43.636 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T17:48:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T17:48:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:48:43.975 INFO:tasks.workunit.client.0.smithi012.stdout:osd.8 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2192011017,v1:127.0.0.1:6867/2192011017] [v2:127.0.0.1:6868/2192011017,v1:127.0.0.1:6869/2192011017] exists,up 72dace4c-9576-4eb8-ab89-9e7c51378f13 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:48:43.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:48:43.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:48:43.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:48:43.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:48:43.980 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:48:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:48:43.982 INFO:tasks.workunit.client.0.smithi012.stdout:add osd9 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:48:43.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 810682a2-ddff-4b14-858b-e45e17b226db' 2024-06-11T17:48:43.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:48:44.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8jWhmb6REABAA4F1UwtOgmO1kiVDKriUmZA== 2024-06-11T17:48:44.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8jWhmb6REABAA4F1UwtOgmO1kiVDKriUmZA=="}' 2024-06-11T17:48:44.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 810682a2-ddff-4b14-858b-e45e17b226db -i td/osd-scrub-repair/9/new.json 2024-06-11T17:48:44.570 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:48:44.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-11T17:48:44.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8jWhmb6REABAA4F1UwtOgmO1kiVDKriUmZA== --osd-uuid 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:48:44.624 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.620+0000 7fceec5647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.704+0000 7fceec5647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:44.707 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.704+0000 7fceec5647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:44.707 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.704+0000 7fceec5647c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:48:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.704+0000 7fceec5647c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:48:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.704+0000 7fceec5647c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:48:44.796 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:44.792+0000 7fceec5647c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-11T17:48:51.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-11T17:48:51.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:48:51.416 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd9 key to auth repository 2024-06-11T17:48:51.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-11T17:48:51.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:48:51.833 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.9 2024-06-11T17:48:51.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-11T17:48:51.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:48:51.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:48:51.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:48:51.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:48:51.866 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:51.864+0000 7f86b805d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:51.904 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:51.900+0000 7f86b805d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:51.906 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:51.904+0000 7f86b805d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:48:52.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:52.143 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:48:52.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:48:52.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:52.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:52.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:52.468+0000 7f86b805d7c0 -1 Falling back to public interface 2024-06-11T17:48:52.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:53.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:53.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:53.505 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:48:53.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:48:53.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:53.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:53.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:54.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:54.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:54.831 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:48:54.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:48:54.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:54.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:55.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:55.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:48:55.308+0000 7f86b805d7c0 -1 osd.9 0 log_to_monitors true 2024-06-11T17:48:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:56.180 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:48:56.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:48:56.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:56.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:56.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:57.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:57.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:57.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:48:57.542 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:48:57.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:57.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:57.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:48:58.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:48:58.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:48:58.890 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:48:58.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:48:58.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:48:58.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:48:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:00.294 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:49:00.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:49:00.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:00.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:00.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:01.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:01.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:01.631 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:49:01.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:49:01.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:01.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:01.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:02.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:02.971 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:49:02.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:49:02.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:02.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:03.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:04.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:04.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:04.312 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:49:04.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:49:04.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:04.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:04.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:05.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:05.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:05.649 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:49:05.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:49:05.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:05.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:05.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:06.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:06.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:06.989 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:49:06.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:49:06.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:06.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:07.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:08.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:08.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:08.327 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:49:08.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:49:08.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:08.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:08.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:09.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:09.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:09.657 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:49:09.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:49:09.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:09.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:09.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:10.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:10.997 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:49:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:49:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:11.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:12.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:12.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:12.333 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:49:12.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:49:12.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:12.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:12.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:13.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:13.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:13.954 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:49:13.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:49:13.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:13.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:14.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:15.575 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:49:15.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:15.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:15.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:49:15.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:15.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:16.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:17.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:17.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:17.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:49:17.207 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:49:17.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:17.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:17.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:18.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:18.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:18.871 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:49:18.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:49:18.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:18.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:19.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:20.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:20.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:20.439 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:49:20.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:49:20.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:20.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:20.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:21.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:21.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:21.945 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:49:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:49:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:21.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:22.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:23.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:23.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:23.280 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:49:23.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:49:23.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:23.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:23.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:24.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:24.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:24.817 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:49:24.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:49:24.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:24.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:49:25.160 INFO:tasks.workunit.client.0.smithi012.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1261650222,v1:127.0.0.1:6875/1261650222] [v2:127.0.0.1:6876/1261650222,v1:127.0.0.1:6877/1261650222] exists,up 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:49:25.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:49:25.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:49:25.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:49:25.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-11T17:49:25.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:49:25.477 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:49:25.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:49:25.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:49:25.936 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:49:25.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:49:26.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:49:32.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-11T17:49:32.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:49:32.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:49:32.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:49:32.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:49:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:49:32.303 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:49:32.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:49:32.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:49:32.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:49:32.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:49:32.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:49:32.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:49:32.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:49:32.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:49:32.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:49:32.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:49:32.777 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:49:32.777 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:49:32.777 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:32.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:49:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836568 2024-06-11T17:49:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836568 2024-06-11T17:49:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568' 2024-06-11T17:49:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:32.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:49:33.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673035 2024-06-11T17:49:33.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673035 2024-06-11T17:49:33.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035' 2024-06-11T17:49:33.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:49:33.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509508 2024-06-11T17:49:33.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509508 2024-06-11T17:49:33.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508' 2024-06-11T17:49:33.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:49:33.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345978 2024-06-11T17:49:33.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345978 2024-06-11T17:49:33.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978' 2024-06-11T17:49:33.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.389 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:49:33.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182454 2024-06-11T17:49:33.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182454 2024-06-11T17:49:33.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454' 2024-06-11T17:49:33.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:49:33.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018921 2024-06-11T17:49:33.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018921 2024-06-11T17:49:33.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454 5-128849018921' 2024-06-11T17:49:33.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.689 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:49:33.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855392 2024-06-11T17:49:33.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855392 2024-06-11T17:49:33.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454 5-128849018921 6-150323855392' 2024-06-11T17:49:33.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:49:33.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691861 2024-06-11T17:49:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691861 2024-06-11T17:49:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454 5-128849018921 6-150323855392 7-171798691861' 2024-06-11T17:49:33.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:33.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:49:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561037 2024-06-11T17:49:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561037 2024-06-11T17:49:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454 5-128849018921 6-150323855392 7-171798691861 8-188978561037' 2024-06-11T17:49:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:34.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:49:34.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397511 2024-06-11T17:49:34.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397511 2024-06-11T17:49:34.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673035 2-64424509508 3-85899345978 4-107374182454 5-128849018921 6-150323855392 7-171798691861 8-188978561037 9-210453397511' 2024-06-11T17:49:34.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:34.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836568 2024-06-11T17:49:34.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:34.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:49:34.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836568 2024-06-11T17:49:34.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:34.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836568 2024-06-11T17:49:34.280 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836568 2024-06-11T17:49:34.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-06-11T17:49:34.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:49:34.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836568 -lt 21474836568 2024-06-11T17:49:34.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:34.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:34.604 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673035 2024-06-11T17:49:34.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:49:34.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673035 2024-06-11T17:49:34.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:34.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673035 2024-06-11T17:49:34.607 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949673035 2024-06-11T17:49:34.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-06-11T17:49:34.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:49:35.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673036 -lt 42949673035 2024-06-11T17:49:35.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:35.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509508 2024-06-11T17:49:35.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:35.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:49:35.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509508 2024-06-11T17:49:35.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:35.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509508 2024-06-11T17:49:35.140 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509508 2024-06-11T17:49:35.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-06-11T17:49:35.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:49:35.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509509 -lt 64424509508 2024-06-11T17:49:35.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:35.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345978 2024-06-11T17:49:35.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:35.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:49:35.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345978 2024-06-11T17:49:35.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:35.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345978 2024-06-11T17:49:35.493 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345978 2024-06-11T17:49:35.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-06-11T17:49:35.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:49:36.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345979 -lt 85899345978 2024-06-11T17:49:36.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:36.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182454 2024-06-11T17:49:36.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:36.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:49:36.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182454 2024-06-11T17:49:36.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:36.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182454 2024-06-11T17:49:36.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-06-11T17:49:36.167 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.4 seq 107374182454 2024-06-11T17:49:36.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:49:36.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182454 -lt 107374182454 2024-06-11T17:49:36.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:36.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018921 2024-06-11T17:49:36.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:36.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:49:36.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018921 2024-06-11T17:49:36.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:36.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018921 2024-06-11T17:49:36.501 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.5 seq 128849018921 2024-06-11T17:49:36.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-06-11T17:49:36.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:49:36.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018921 -lt 128849018921 2024-06-11T17:49:36.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:36.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855392 2024-06-11T17:49:36.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:36.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:49:36.876 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855392 2024-06-11T17:49:36.876 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:36.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855392 2024-06-11T17:49:36.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-06-11T17:49:36.878 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.6 seq 150323855392 2024-06-11T17:49:36.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:49:37.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855393 -lt 150323855392 2024-06-11T17:49:37.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:37.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691861 2024-06-11T17:49:37.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:37.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:49:37.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:37.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691861 2024-06-11T17:49:37.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691861 2024-06-11T17:49:37.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-06-11T17:49:37.215 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.7 seq 171798691861 2024-06-11T17:49:37.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:49:37.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691861 -lt 171798691861 2024-06-11T17:49:37.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:37.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561037 2024-06-11T17:49:37.540 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:37.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:49:37.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:37.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561037 2024-06-11T17:49:37.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561037 2024-06-11T17:49:37.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561037' 2024-06-11T17:49:37.545 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.8 seq 188978561037 2024-06-11T17:49:37.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:49:37.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561037 -lt 188978561037 2024-06-11T17:49:37.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:37.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-210453397511 2024-06-11T17:49:37.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:37.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:49:37.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-210453397511 2024-06-11T17:49:37.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:37.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397511 2024-06-11T17:49:37.866 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.9 seq 210453397511 2024-06-11T17:49:37.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 210453397511' 2024-06-11T17:49:37.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:49:38.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397512 -lt 210453397511 2024-06-11T17:49:38.195 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:49:38.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:49:38.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:49:38.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:49:38.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:49:38.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:49:38.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:49:38.995 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:49:38.995 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:49:38.995 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:49:39.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:49:39.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:49:39.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:49:39.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-06-11T17:49:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:49:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:49:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T17:49:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:49:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-11T17:49:39.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:49:39.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:49:40.747 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:49:40.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:49:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T17:49:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:49:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:49:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:49:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:49:41.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:49:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:49:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:49:41.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:49:41.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:49:41.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:49:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:49:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:49:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:49:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:49:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:49:41.916 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:49:42.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:49:42.372 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:49:42.372 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:49:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:42.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:49:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836571 2024-06-11T17:49:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836571 2024-06-11T17:49:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571' 2024-06-11T17:49:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:42.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:49:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673038 2024-06-11T17:49:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673038 2024-06-11T17:49:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038' 2024-06-11T17:49:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:42.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:49:42.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509511 2024-06-11T17:49:42.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509511 2024-06-11T17:49:42.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511' 2024-06-11T17:49:42.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:42.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:49:43.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345981 2024-06-11T17:49:43.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345981 2024-06-11T17:49:43.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981' 2024-06-11T17:49:43.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:49:43.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182457 2024-06-11T17:49:43.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182457 2024-06-11T17:49:43.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457' 2024-06-11T17:49:43.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:49:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018924 2024-06-11T17:49:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018924 2024-06-11T17:49:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457 5-128849018924' 2024-06-11T17:49:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:49:43.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855395 2024-06-11T17:49:43.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855395 2024-06-11T17:49:43.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457 5-128849018924 6-150323855395' 2024-06-11T17:49:43.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.690 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:49:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691864 2024-06-11T17:49:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691864 2024-06-11T17:49:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457 5-128849018924 6-150323855395 7-171798691864' 2024-06-11T17:49:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:49:43.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561040 2024-06-11T17:49:43.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561040 2024-06-11T17:49:43.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457 5-128849018924 6-150323855395 7-171798691864 8-188978561040' 2024-06-11T17:49:43.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:49:43.997 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:49:44.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397514 2024-06-11T17:49:44.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397514 2024-06-11T17:49:44.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509511 3-85899345981 4-107374182457 5-128849018924 6-150323855395 7-171798691864 8-188978561040 9-210453397514' 2024-06-11T17:49:44.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:44.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836571 2024-06-11T17:49:44.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:44.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:49:44.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836571 2024-06-11T17:49:44.150 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:44.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836571 2024-06-11T17:49:44.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-06-11T17:49:44.150 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836571 2024-06-11T17:49:44.150 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:49:44.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836571 -lt 21474836571 2024-06-11T17:49:44.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:44.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:44.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673038 2024-06-11T17:49:44.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:49:44.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673038 2024-06-11T17:49:44.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:44.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673038 2024-06-11T17:49:44.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-06-11T17:49:44.479 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949673038 2024-06-11T17:49:44.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:49:44.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673038 -lt 42949673038 2024-06-11T17:49:44.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:44.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509511 2024-06-11T17:49:44.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:44.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:49:44.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509511 2024-06-11T17:49:44.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509511 2024-06-11T17:49:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-06-11T17:49:44.837 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509511 2024-06-11T17:49:44.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:49:45.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509511 -lt 64424509511 2024-06-11T17:49:45.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:45.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345981 2024-06-11T17:49:45.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:45.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:49:45.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345981 2024-06-11T17:49:45.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:45.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345981 2024-06-11T17:49:45.258 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345981 2024-06-11T17:49:45.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345981' 2024-06-11T17:49:45.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:49:45.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345982 -lt 85899345981 2024-06-11T17:49:45.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:45.686 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182457 2024-06-11T17:49:45.687 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:45.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:49:45.690 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182457 2024-06-11T17:49:45.690 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:45.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182457 2024-06-11T17:49:45.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-06-11T17:49:45.692 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.4 seq 107374182457 2024-06-11T17:49:45.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:49:46.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182457 -lt 107374182457 2024-06-11T17:49:46.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:46.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018924 2024-06-11T17:49:46.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:46.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:49:46.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018924 2024-06-11T17:49:46.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:46.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018924 2024-06-11T17:49:46.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-06-11T17:49:46.039 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.5 seq 128849018924 2024-06-11T17:49:46.039 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:49:46.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018924 -lt 128849018924 2024-06-11T17:49:46.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:46.402 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855395 2024-06-11T17:49:46.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:46.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:49:46.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855395 2024-06-11T17:49:46.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:46.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855395 2024-06-11T17:49:46.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-06-11T17:49:46.407 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.6 seq 150323855395 2024-06-11T17:49:46.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:49:46.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855396 -lt 150323855395 2024-06-11T17:49:46.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:46.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691864 2024-06-11T17:49:46.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:46.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:49:46.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691864 2024-06-11T17:49:46.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:46.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691864 2024-06-11T17:49:46.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-06-11T17:49:46.800 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.7 seq 171798691864 2024-06-11T17:49:46.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:49:47.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691864 -lt 171798691864 2024-06-11T17:49:47.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:47.199 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561040 2024-06-11T17:49:47.199 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:47.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:49:47.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561040 2024-06-11T17:49:47.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:47.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561040 2024-06-11T17:49:47.204 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.8 seq 188978561040 2024-06-11T17:49:47.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561040' 2024-06-11T17:49:47.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:49:47.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561040 -lt 188978561040 2024-06-11T17:49:47.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:49:47.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-210453397514 2024-06-11T17:49:47.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:49:47.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:49:47.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-210453397514 2024-06-11T17:49:47.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:49:47.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397514 2024-06-11T17:49:47.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 210453397514' 2024-06-11T17:49:47.570 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.9 seq 210453397514 2024-06-11T17:49:47.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:49:47.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397515 -lt 210453397514 2024-06-11T17:49:47.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:49:47.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:49:47.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:49:48.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:49:48.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:49:48.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:49:48.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:49:48.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:49:48.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:49:48.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:49:49.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:49:49.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:49:49.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T17:49:49.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T17:49:49.586 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:49:49.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T17:49:50.215 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:49:50.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T17:49:50.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T17:49:50.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:49:50.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-11T17:49:50.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:49:50.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:49:50.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:49:50.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:49:51.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-11T17:49:51.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-11T17:49:51.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-11T17:49:51.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T17:49:51.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-11T17:49:51.556 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:49:51.556 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T17:49:51.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:9 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr:7' 2024-06-11T17:49:51.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-11T17:49:51.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-11T17:49:51.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:49:51.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:49:51.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:49:51.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:49:51.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:49:51.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:49:53.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:49:53.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:49:55.438 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:49:55.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:49:55.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:49:55.983 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:49:55.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:49:55.985 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:49:55.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:49:56.019 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:49:56.015+0000 7f0c123b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:49:56.022 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:49:56.019+0000 7f0c123b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:49:56.023 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:49:56.019+0000 7f0c123b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:49:56.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:49:56.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:56.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:49:56.556 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:49:56.551+0000 7f0c123b57c0 -1 Falling back to public interface 2024-06-11T17:49:56.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:49:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:49:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:49:58.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:59.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:49:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:49:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:49:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:49:59.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:49:59.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:49:59.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:49:59.520 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:49:59.515+0000 7f0c123b57c0 -1 osd.3 67 log_to_monitors true 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:00.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:50:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/4077581000,v1:127.0.0.1:6827/4077581000] [v2:127.0.0.1:6828/4077581000,v1:127.0.0.1:6829/4077581000] exists,up 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:50:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:50:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:50:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:50:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:50:01.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:50:01.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:50:01.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:50:01.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:50:01.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:50:01.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:50:01.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:50:01.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:50:01.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:50:01.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:50:01.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:50:01.870 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:01.870 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:01.870 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:01.870 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:01.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:50:02.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836576 2024-06-11T17:50:02.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836576 2024-06-11T17:50:02.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576' 2024-06-11T17:50:02.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:50:02.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673043 2024-06-11T17:50:02.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673043 2024-06-11T17:50:02.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043' 2024-06-11T17:50:02.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:50:02.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509516 2024-06-11T17:50:02.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509516 2024-06-11T17:50:02.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516' 2024-06-11T17:50:02.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:50:02.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678018 2024-06-11T17:50:02.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678018 2024-06-11T17:50:02.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018' 2024-06-11T17:50:02.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:50:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182461 2024-06-11T17:50:02.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182461 2024-06-11T17:50:02.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461' 2024-06-11T17:50:02.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:50:02.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018929 2024-06-11T17:50:02.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018929 2024-06-11T17:50:02.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461 5-128849018929' 2024-06-11T17:50:02.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:02.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:50:03.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855400 2024-06-11T17:50:03.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855400 2024-06-11T17:50:03.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461 5-128849018929 6-150323855400' 2024-06-11T17:50:03.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:03.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:50:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691868 2024-06-11T17:50:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691868 2024-06-11T17:50:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461 5-128849018929 6-150323855400 7-171798691868' 2024-06-11T17:50:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:03.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:50:03.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561044 2024-06-11T17:50:03.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561044 2024-06-11T17:50:03.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461 5-128849018929 6-150323855400 7-171798691868 8-188978561044' 2024-06-11T17:50:03.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:03.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:50:03.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397519 2024-06-11T17:50:03.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397519 2024-06-11T17:50:03.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-304942678018 4-107374182461 5-128849018929 6-150323855400 7-171798691868 8-188978561044 9-210453397519' 2024-06-11T17:50:03.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:03.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836576 2024-06-11T17:50:03.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:03.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:50:03.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836576 2024-06-11T17:50:03.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:03.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836576 2024-06-11T17:50:03.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-06-11T17:50:03.463 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836576 2024-06-11T17:50:03.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:50:03.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836576 -lt 21474836576 2024-06-11T17:50:03.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:03.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:03.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673043 2024-06-11T17:50:03.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:50:03.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:03.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673043 2024-06-11T17:50:03.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673043 2024-06-11T17:50:03.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-06-11T17:50:03.790 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673043 2024-06-11T17:50:03.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:50:04.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673043 -lt 42949673043 2024-06-11T17:50:04.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:04.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509516 2024-06-11T17:50:04.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:04.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:50:04.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509516 2024-06-11T17:50:04.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:04.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509516 2024-06-11T17:50:04.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-06-11T17:50:04.109 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509516 2024-06-11T17:50:04.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:50:04.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509516 -lt 64424509516 2024-06-11T17:50:04.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:04.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678018 2024-06-11T17:50:04.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:04.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:50:04.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678018 2024-06-11T17:50:04.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:04.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678018 2024-06-11T17:50:04.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-06-11T17:50:04.481 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 304942678018 2024-06-11T17:50:04.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:50:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678018 -lt 304942678018 2024-06-11T17:50:04.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:04.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:04.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182461 2024-06-11T17:50:04.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:50:04.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182461 2024-06-11T17:50:04.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:04.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182461 2024-06-11T17:50:04.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-06-11T17:50:04.891 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182461 2024-06-11T17:50:04.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:50:05.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182462 -lt 107374182461 2024-06-11T17:50:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:05.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018929 2024-06-11T17:50:05.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:05.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:50:05.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018929 2024-06-11T17:50:05.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:05.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018929 2024-06-11T17:50:05.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018929' 2024-06-11T17:50:05.295 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 128849018929 2024-06-11T17:50:05.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:50:05.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018929 -lt 128849018929 2024-06-11T17:50:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:05.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855400 2024-06-11T17:50:05.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:05.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:50:05.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855400 2024-06-11T17:50:05.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855400 2024-06-11T17:50:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-06-11T17:50:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855400 2024-06-11T17:50:05.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:50:06.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855401 -lt 150323855400 2024-06-11T17:50:06.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:06.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691868 2024-06-11T17:50:06.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:06.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:50:06.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691868 2024-06-11T17:50:06.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:06.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691868 2024-06-11T17:50:06.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-06-11T17:50:06.140 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691868 2024-06-11T17:50:06.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:50:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691869 -lt 171798691868 2024-06-11T17:50:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:06.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561044 2024-06-11T17:50:06.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:06.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:50:06.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561044 2024-06-11T17:50:06.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:06.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561044 2024-06-11T17:50:06.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561044' 2024-06-11T17:50:06.491 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 188978561044 2024-06-11T17:50:06.492 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:50:06.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561045 -lt 188978561044 2024-06-11T17:50:06.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:06.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-210453397519 2024-06-11T17:50:06.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:06.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:50:06.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-210453397519 2024-06-11T17:50:06.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397519 2024-06-11T17:50:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 210453397519' 2024-06-11T17:50:06.854 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 210453397519 2024-06-11T17:50:06.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:50:07.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397520 -lt 210453397519 2024-06-11T17:50:07.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:50:07.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:07.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:07.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:50:07.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:50:07.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:50:07.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:50:07.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:50:07.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:50:07.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:50:07.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:50:08.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:50:08.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:50:08.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:08.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:08.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:50:08.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:50:08.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:50:08.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:50:08.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:50:08.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:50:08.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:50:08.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:50:08.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:50:08.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:50:08.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:50:08.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:50:08.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:08.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:08.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:08.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:09.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:09.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:50:09.622 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:09.638 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:09.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:09.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:09.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:09.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:49:40.579770+0000 '>' 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:09.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:10.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:10.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:10.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:10.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:10.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:10.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:10.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:11.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:49:40.579770+0000 '>' 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:11.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:12.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:12.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:12.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:12.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:12.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:12.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:12.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:12.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:49:40.579770+0000 '>' 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:12.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:13.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:13.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:13.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:13.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:13.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:13.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:13.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:14.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:49:40.579770+0000 '>' 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:14.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:15.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:15.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:15.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:15.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:15.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:15.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:15.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:15.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:10.582498+0000 '>' 2024-06-11T17:49:40.579770+0000 2024-06-11T17:50:15.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:50:15.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T17:50:15.377 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:50:15.377 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:50:15.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:50:15.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:50:15.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:50:15.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:50:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T17:50:15.981 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:50:15.982 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:50:15.982 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:50:16.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T17:50:16.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:50:16.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:50:16.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T17:50:16.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:50:16.519 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:50:16.520 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:50:16.520 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:50:16.521 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:16.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:16.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:50:16.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:50:16.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T17:50:16.523 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.9 2024-06-11T17:50:16.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:50:16.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T17:50:16.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T17:50:16.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:50:16.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:50:16.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:50:16.562 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:16.559+0000 7f22ee5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:16.901 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:16.899+0000 7f22ee5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:16.904 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:16.899+0000 7f22ee5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:17.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:17.468 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:17.463+0000 7f22ee5f27c0 -1 Falling back to public interface 2024-06-11T17:50:17.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:18.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:18.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:18.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:50:18.835 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:18.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:18.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:19.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:20.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:20.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:20.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:50:20.169 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:20.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:20.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:20.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:20.311+0000 7f22ee5f27c0 -1 osd.9 72 log_to_monitors true 2024-06-11T17:50:20.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:50:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:21.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:21.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [49,72) [v2:127.0.0.1:6874/525537547,v1:127.0.0.1:6875/525537547] [v2:127.0.0.1:6876/525537547,v1:127.0.0.1:6877/525537547] exists,up 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:50:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:50:22.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:50:22.045 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:50:22.046 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:50:22.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:50:22.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:50:22.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:50:22.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:50:22.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:50:22.554 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:50:22.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:22.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:50:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836581 2024-06-11T17:50:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836581 2024-06-11T17:50:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581' 2024-06-11T17:50:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:22.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:50:22.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673048 2024-06-11T17:50:22.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673048 2024-06-11T17:50:22.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048' 2024-06-11T17:50:22.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:22.860 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:50:23.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509521 2024-06-11T17:50:23.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509521 2024-06-11T17:50:23.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521' 2024-06-11T17:50:23.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.173 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:50:23.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678023 2024-06-11T17:50:23.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678023 2024-06-11T17:50:23.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023' 2024-06-11T17:50:23.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:50:23.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182467 2024-06-11T17:50:23.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182467 2024-06-11T17:50:23.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467' 2024-06-11T17:50:23.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.504 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:50:23.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018934 2024-06-11T17:50:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018934 2024-06-11T17:50:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467 5-128849018934' 2024-06-11T17:50:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:50:23.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855405 2024-06-11T17:50:23.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855405 2024-06-11T17:50:23.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467 5-128849018934 6-150323855405' 2024-06-11T17:50:23.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:50:23.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691874 2024-06-11T17:50:23.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691874 2024-06-11T17:50:23.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467 5-128849018934 6-150323855405 7-171798691874' 2024-06-11T17:50:23.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:23.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:50:24.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561050 2024-06-11T17:50:24.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561050 2024-06-11T17:50:24.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467 5-128849018934 6-150323855405 7-171798691874 8-188978561050' 2024-06-11T17:50:24.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:24.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:50:24.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=330712481794 2024-06-11T17:50:24.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 330712481794 2024-06-11T17:50:24.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509521 3-304942678023 4-107374182467 5-128849018934 6-150323855405 7-171798691874 8-188978561050 9-330712481794' 2024-06-11T17:50:24.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:24.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836581 2024-06-11T17:50:24.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:24.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:50:24.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836581 2024-06-11T17:50:24.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:24.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836581 2024-06-11T17:50:24.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-06-11T17:50:24.237 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836581 2024-06-11T17:50:24.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:50:24.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836581 -lt 21474836581 2024-06-11T17:50:24.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:24.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:24.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673048 2024-06-11T17:50:24.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:50:24.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673048 2024-06-11T17:50:24.558 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673048 2024-06-11T17:50:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-06-11T17:50:24.560 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673048 2024-06-11T17:50:24.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:50:24.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673048 -lt 42949673048 2024-06-11T17:50:24.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:24.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:24.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509521 2024-06-11T17:50:24.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:50:24.922 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509521 2024-06-11T17:50:24.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:24.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509521 2024-06-11T17:50:24.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-06-11T17:50:24.925 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509521 2024-06-11T17:50:24.925 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:50:25.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509522 -lt 64424509521 2024-06-11T17:50:25.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:25.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:25.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678023 2024-06-11T17:50:25.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:50:25.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678023 2024-06-11T17:50:25.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:25.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678023 2024-06-11T17:50:25.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-06-11T17:50:25.350 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 304942678023 2024-06-11T17:50:25.350 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:50:25.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678024 -lt 304942678023 2024-06-11T17:50:25.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:25.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182467 2024-06-11T17:50:25.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:25.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:50:25.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182467 2024-06-11T17:50:25.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:25.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182467 2024-06-11T17:50:25.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-06-11T17:50:25.711 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182467 2024-06-11T17:50:25.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:50:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182467 -lt 107374182467 2024-06-11T17:50:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:26.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018934 2024-06-11T17:50:26.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:26.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:50:26.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018934 2024-06-11T17:50:26.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:26.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018934 2024-06-11T17:50:26.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-06-11T17:50:26.033 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 128849018934 2024-06-11T17:50:26.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:50:26.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018934 -lt 128849018934 2024-06-11T17:50:26.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:26.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855405 2024-06-11T17:50:26.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:26.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:50:26.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855405 2024-06-11T17:50:26.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:26.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855405 2024-06-11T17:50:26.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-06-11T17:50:26.370 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855405 2024-06-11T17:50:26.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:50:26.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855406 -lt 150323855405 2024-06-11T17:50:26.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:26.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691874 2024-06-11T17:50:26.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:26.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:50:26.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691874 2024-06-11T17:50:26.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:26.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691874 2024-06-11T17:50:26.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-06-11T17:50:26.732 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691874 2024-06-11T17:50:26.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:50:27.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691874 -lt 171798691874 2024-06-11T17:50:27.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:27.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561050 2024-06-11T17:50:27.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:27.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:50:27.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561050 2024-06-11T17:50:27.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:27.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561050 2024-06-11T17:50:27.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561050' 2024-06-11T17:50:27.134 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 188978561050 2024-06-11T17:50:27.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:50:27.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561050 -lt 188978561050 2024-06-11T17:50:27.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:27.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-330712481794 2024-06-11T17:50:27.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:50:27.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:27.490 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-330712481794 2024-06-11T17:50:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=330712481794 2024-06-11T17:50:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-06-11T17:50:27.491 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 330712481794 2024-06-11T17:50:27.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:50:27.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 330712481795 -lt 330712481794 2024-06-11T17:50:27.842 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:50:27.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:27.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:28.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:50:28.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:50:28.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:50:28.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:50:28.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:50:28.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:28.651 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:29.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:50:29.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:50:29.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:50:29.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:50:29.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:50:29.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-11T17:50:29.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:50:29.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:50:29.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-11T17:50:29.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:50:29.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:50:29.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:50:29.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:50:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:50:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:50:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T17:50:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:50:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:50:29.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T17:50:32.368 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:50:32.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:50:32.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:50:32.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:50:32.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:50:32.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:50:32.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:50:32.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:50:32.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:50:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:50:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T17:50:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.5 2024-06-11T17:50:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:50:32.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T17:50:32.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T17:50:32.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:50:32.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:50:32.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:50:32.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:32.939+0000 7f491b66f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:32.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:32.943+0000 7f491b66f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:32.947 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:32.943+0000 7f491b66f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:50:33.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:33.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:50:33.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:33.495+0000 7f491b66f7c0 -1 Falling back to public interface 2024-06-11T17:50:33.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:34.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:34.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:34.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:50:34.654 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:34.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:34.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:50:34.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:50:35.991 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:35.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:35.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:50:36.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:36.307+0000 7f491b66f7c0 -1 osd.5 78 log_to_monitors true 2024-06-11T17:50:36.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:37.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:37.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:37.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:50:37.329 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:37.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:37.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:osd.5 up in weight 1 up_from 82 up_thru 68 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/3017612209,v1:127.0.0.1:6843/3017612209] [v2:127.0.0.1:6844/3017612209,v1:127.0.0.1:6845/3017612209] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:50:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:50:37.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:50:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:50:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:50:37.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:50:37.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:50:37.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:50:37.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:50:38.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:50:38.431 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:38.431 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:38.431 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:38.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:50:38.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836585 2024-06-11T17:50:38.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836585 2024-06-11T17:50:38.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585' 2024-06-11T17:50:38.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:38.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:50:39.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673052 2024-06-11T17:50:39.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673052 2024-06-11T17:50:39.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052' 2024-06-11T17:50:39.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:39.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:50:39.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509525 2024-06-11T17:50:39.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509525 2024-06-11T17:50:39.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525' 2024-06-11T17:50:39.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:39.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:50:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678028 2024-06-11T17:50:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678028 2024-06-11T17:50:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028' 2024-06-11T17:50:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:39.984 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:50:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182471 2024-06-11T17:50:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182471 2024-06-11T17:50:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471' 2024-06-11T17:50:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:40.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:50:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=352187318274 2024-06-11T17:50:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 352187318274 2024-06-11T17:50:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471 5-352187318274' 2024-06-11T17:50:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:40.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:50:40.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855410 2024-06-11T17:50:40.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855410 2024-06-11T17:50:40.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471 5-352187318274 6-150323855410' 2024-06-11T17:50:40.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:40.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:50:40.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691878 2024-06-11T17:50:40.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691878 2024-06-11T17:50:40.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471 5-352187318274 6-150323855410 7-171798691878' 2024-06-11T17:50:40.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:40.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:50:40.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561054 2024-06-11T17:50:40.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561054 2024-06-11T17:50:40.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471 5-352187318274 6-150323855410 7-171798691878 8-188978561054' 2024-06-11T17:50:40.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:40.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:50:40.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=330712481799 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 330712481799 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673052 2-64424509525 3-304942678028 4-107374182471 5-352187318274 6-150323855410 7-171798691878 8-188978561054 9-330712481799' 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836585 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:40.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:50:40.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836585 2024-06-11T17:50:40.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:40.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836585 2024-06-11T17:50:40.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-06-11T17:50:40.921 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836585 2024-06-11T17:50:40.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:50:41.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836586 -lt 21474836585 2024-06-11T17:50:41.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:41.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673052 2024-06-11T17:50:41.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:41.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:50:41.267 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673052 2024-06-11T17:50:41.267 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:41.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673052 2024-06-11T17:50:41.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-06-11T17:50:41.269 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673052 2024-06-11T17:50:41.269 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:50:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673053 -lt 42949673052 2024-06-11T17:50:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:41.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:41.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509525 2024-06-11T17:50:41.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:50:41.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509525 2024-06-11T17:50:41.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:41.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509525 2024-06-11T17:50:41.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-06-11T17:50:41.615 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509525 2024-06-11T17:50:41.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:50:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509526 -lt 64424509525 2024-06-11T17:50:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:41.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678028 2024-06-11T17:50:41.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:41.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:50:41.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678028 2024-06-11T17:50:41.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:41.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678028 2024-06-11T17:50:41.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678028' 2024-06-11T17:50:41.981 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 304942678028 2024-06-11T17:50:41.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:50:42.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678028 -lt 304942678028 2024-06-11T17:50:42.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:42.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182471 2024-06-11T17:50:42.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:42.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:50:42.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182471 2024-06-11T17:50:42.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:42.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182471 2024-06-11T17:50:42.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-06-11T17:50:42.367 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182471 2024-06-11T17:50:42.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:50:42.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182471 -lt 107374182471 2024-06-11T17:50:42.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:42.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-352187318274 2024-06-11T17:50:42.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:42.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:50:42.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-352187318274 2024-06-11T17:50:42.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=352187318274 2024-06-11T17:50:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-06-11T17:50:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 352187318274 2024-06-11T17:50:42.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:50:43.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 352187318274 -lt 352187318274 2024-06-11T17:50:43.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:43.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855410 2024-06-11T17:50:43.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:43.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:50:43.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855410 2024-06-11T17:50:43.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:43.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855410 2024-06-11T17:50:43.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-06-11T17:50:43.095 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855410 2024-06-11T17:50:43.095 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:50:43.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855410 -lt 150323855410 2024-06-11T17:50:43.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:43.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691878 2024-06-11T17:50:43.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:43.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:50:43.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691878 2024-06-11T17:50:43.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:43.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691878 2024-06-11T17:50:43.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-06-11T17:50:43.482 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691878 2024-06-11T17:50:43.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:50:43.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691878 -lt 171798691878 2024-06-11T17:50:43.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:43.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561054 2024-06-11T17:50:43.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:43.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:50:43.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561054 2024-06-11T17:50:43.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:43.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561054 2024-06-11T17:50:43.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561054' 2024-06-11T17:50:43.854 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 188978561054 2024-06-11T17:50:43.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:50:44.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561054 -lt 188978561054 2024-06-11T17:50:44.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:50:44.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-330712481799 2024-06-11T17:50:44.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:50:44.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:50:44.218 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-330712481799 2024-06-11T17:50:44.218 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:50:44.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=330712481799 2024-06-11T17:50:44.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 330712481799' 2024-06-11T17:50:44.219 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 330712481799 2024-06-11T17:50:44.220 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:50:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 330712481799 -lt 330712481799 2024-06-11T17:50:44.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:50:44.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:44.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:45.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:50:45.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:50:45.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:50:45.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:50:45.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:50:45.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:50:45.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:50:45.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:50:45.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:50:45.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:50:45.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T17:50:45.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:50:45.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:50:45.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:50:45.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:50:46.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T17:50:46.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T17:50:46.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:50:46.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:50:46.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:46.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:46.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:46.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:46.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:46.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:50:46.706 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:50:46.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:46.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:50:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:50:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:50:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:46.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:46.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:46.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:46.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:46.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:47.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:10.582498+0000 '>' 2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:47.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:48.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:48.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:48.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:48.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:48.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:48.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:48.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:48.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:10.582498+0000 '>' 2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:48.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:49.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:49.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:49.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:49.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:49.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:49.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:49.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:49.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:10.582498+0000 '>' 2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:49.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:50:50.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:50:50.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:50:50.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:50:50.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:50:50.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:50:50.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:50:50.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:50:51.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:10.582498+0000 2024-06-11T17:50:51.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:50:51.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:50:51.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:50:51.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:50:51.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T17:50:52.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:50:52.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:50:52.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T17:50:52.565 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T17:50:52.565 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T17:50:52.565 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:50:52.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:50:52.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:50:52.865 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:50:52.865 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:50:52.865 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:50:52.865 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:52.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T17:50:52.866 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.9 2024-06-11T17:50:52.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:50:52.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T17:50:52.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T17:50:52.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:50:52.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:50:52.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:50:52.902 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:52.899+0000 7f1ab5fb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:52.990 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:52.987+0000 7f1ab5fb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:52.992 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:52.987+0000 7f1ab5fb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:53.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:53.536 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:53.531+0000 7f1ab5fb37c0 -1 Falling back to public interface 2024-06-11T17:50:53.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:54.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:54.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:54.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:50:54.765 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:54.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:54.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:55.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:56.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:56.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:56.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:50:56.104 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:56.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:56.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:56.226 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:50:56.223+0000 7f1ab5fb37c0 -1 osd.9 84 log_to_monitors true 2024-06-11T17:50:56.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:50:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:50:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:50:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:50:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:50:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/1523053526,v1:127.0.0.1:6875/1523053526] [v2:127.0.0.1:6876/1523053526,v1:127.0.0.1:6877/1523053526] exists,up 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:50:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:50:58.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:50:58.027 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:50:58.027 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:50:58.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:50:58.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:50:58.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:50:58.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:50:58.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:50:58.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:50:58.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:50:58.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:50:58.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:50:58.664 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:58.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:50:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836590 2024-06-11T17:50:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836590 2024-06-11T17:50:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590' 2024-06-11T17:50:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:58.819 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:50:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673057 2024-06-11T17:50:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673057 2024-06-11T17:50:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057' 2024-06-11T17:50:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:50:59.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509530 2024-06-11T17:50:59.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509530 2024-06-11T17:50:59.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530' 2024-06-11T17:50:59.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:50:59.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678032 2024-06-11T17:50:59.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678032 2024-06-11T17:50:59.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032' 2024-06-11T17:50:59.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:50:59.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182476 2024-06-11T17:50:59.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182476 2024-06-11T17:50:59.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476' 2024-06-11T17:50:59.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:50:59.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=352187318279 2024-06-11T17:50:59.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 352187318279 2024-06-11T17:50:59.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476 5-352187318279' 2024-06-11T17:50:59.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:50:59.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855415 2024-06-11T17:50:59.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855415 2024-06-11T17:50:59.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476 5-352187318279 6-150323855415' 2024-06-11T17:50:59.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:50:59.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:51:00.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691883 2024-06-11T17:51:00.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691883 2024-06-11T17:51:00.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476 5-352187318279 6-150323855415 7-171798691883' 2024-06-11T17:51:00.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:00.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:51:00.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561059 2024-06-11T17:51:00.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561059 2024-06-11T17:51:00.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476 5-352187318279 6-150323855415 7-171798691883 8-188978561059' 2024-06-11T17:51:00.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:00.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:51:00.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=377957122050 2024-06-11T17:51:00.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 377957122050 2024-06-11T17:51:00.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836590 1-42949673057 2-64424509530 3-304942678032 4-107374182476 5-352187318279 6-150323855415 7-171798691883 8-188978561059 9-377957122050' 2024-06-11T17:51:00.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:00.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836590 2024-06-11T17:51:00.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:00.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:51:00.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836590 2024-06-11T17:51:00.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:00.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836590 2024-06-11T17:51:00.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-06-11T17:51:00.327 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836590 2024-06-11T17:51:00.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:51:00.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836590 -lt 21474836590 2024-06-11T17:51:00.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:00.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673057 2024-06-11T17:51:00.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:00.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:51:00.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673057 2024-06-11T17:51:00.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:00.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673057 2024-06-11T17:51:00.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-06-11T17:51:00.648 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673057 2024-06-11T17:51:00.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:51:01.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673056 -lt 42949673057 2024-06-11T17:51:01.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T17:51:02.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T17:51:02.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:51:02.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673058 -lt 42949673057 2024-06-11T17:51:02.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:02.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:02.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509530 2024-06-11T17:51:02.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:51:02.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:02.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509530 2024-06-11T17:51:02.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509530 2024-06-11T17:51:02.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-06-11T17:51:02.370 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509530 2024-06-11T17:51:02.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:51:02.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509531 -lt 64424509530 2024-06-11T17:51:02.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:02.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:02.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678032 2024-06-11T17:51:02.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:51:02.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678032 2024-06-11T17:51:02.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678032 2024-06-11T17:51:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678032' 2024-06-11T17:51:02.742 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 304942678032 2024-06-11T17:51:02.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:51:03.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678033 -lt 304942678032 2024-06-11T17:51:03.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:03.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182476 2024-06-11T17:51:03.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:03.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182476 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182476 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182476 2024-06-11T17:51:03.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:51:03.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182476 -lt 107374182476 2024-06-11T17:51:03.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:03.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-352187318279 2024-06-11T17:51:03.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:03.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:51:03.440 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-352187318279 2024-06-11T17:51:03.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:03.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=352187318279 2024-06-11T17:51:03.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2024-06-11T17:51:03.442 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 352187318279 2024-06-11T17:51:03.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:51:03.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 352187318280 -lt 352187318279 2024-06-11T17:51:03.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:03.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855415 2024-06-11T17:51:03.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:03.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:51:03.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855415 2024-06-11T17:51:03.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:03.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855415 2024-06-11T17:51:03.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-06-11T17:51:03.797 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855415 2024-06-11T17:51:03.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:51:04.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855415 -lt 150323855415 2024-06-11T17:51:04.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:04.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691883 2024-06-11T17:51:04.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:04.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:51:04.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691883 2024-06-11T17:51:04.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:04.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691883 2024-06-11T17:51:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-06-11T17:51:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691883 2024-06-11T17:51:04.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:51:04.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691883 -lt 171798691883 2024-06-11T17:51:04.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:04.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561059 2024-06-11T17:51:04.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:04.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:51:04.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561059 2024-06-11T17:51:04.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:04.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561059 2024-06-11T17:51:04.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561059' 2024-06-11T17:51:04.552 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 188978561059 2024-06-11T17:51:04.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:51:04.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561059 -lt 188978561059 2024-06-11T17:51:04.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:04.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:04.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-377957122050 2024-06-11T17:51:04.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:51:04.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-377957122050 2024-06-11T17:51:04.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:04.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=377957122050 2024-06-11T17:51:04.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-06-11T17:51:04.925 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 377957122050 2024-06-11T17:51:04.926 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:51:05.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 377957122051 -lt 377957122050 2024-06-11T17:51:05.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:51:05.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:05.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:05.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:51:05.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:51:05.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:51:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:51:05.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:51:05.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:05.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:06.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:51:06.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:51:06.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:51:06.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:51:06.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 219326"' 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 219326' 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 219328"' 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 219328' 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:06.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 219326 219328' 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 219326 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/219329: /' 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T17:51:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/219331: /' 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:10.576 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:10.577 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:10.579 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:10.582 INFO:tasks.workunit.client.0.smithi012.stderr:219329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:10.583 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:10.584 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:51:10.584 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T17:51:10.584 INFO:tasks.workunit.client.0.smithi012.stderr:219329: start osd.5 2024-06-11T17:51:10.584 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.633 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:10.635 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:10.637 INFO:tasks.workunit.client.0.smithi012.stderr:219331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:10.638 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:51:10.639 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T17:51:10.639 INFO:tasks.workunit.client.0.smithi012.stderr:219331: start osd.9 2024-06-11T17:51:10.639 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:51:15.323 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T17:51:15.323 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T17:51:15.323 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:15.323 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2024-06-11T17:51:10.619+0000 7fbdb2ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2024-06-11T17:51:10.619+0000 7fbdb2ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2024-06-11T17:51:10.623+0000 7fbdb2ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.324 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 0 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2024-06-11T17:51:11.159+0000 7fbdb2ee47c0 -1 Falling back to public interface 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 1 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:15.325 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2024-06-11T17:51:14.363+0000 7fbdb2ee47c0 -1 osd.5 89 log_to_monitors true 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 3 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:15.327 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2024-06-11T17:51:10.659+0000 7f3b450db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2024-06-11T17:51:10.663+0000 7f3b450db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2024-06-11T17:51:10.663+0000 7f3b450db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:15.328 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 0 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2024-06-11T17:51:11.235+0000 7f3b450db7c0 -1 Falling back to public interface 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 1 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.329 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2024-06-11T17:51:14.567+0000 7f3b450db7c0 -1 osd.9 89 log_to_monitors true 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:15.330 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 3 2024-06-11T17:51:15.331 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:15.331 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 4 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:18.047 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: osd.5 up in weight 1 up_from 94 up_thru 94 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6842/1026545995,v1:127.0.0.1:6843/1026545995] [v2:127.0.0.1:6844/1026545995,v1:127.0.0.1:6845/1026545995] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:51:18.048 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:51:18.049 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 1 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 2 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 3 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 4 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 5 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 6 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 7 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 8 2024-06-11T17:51:18.050 INFO:tasks.workunit.client.0.smithi012.stderr:219329: 9' 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836596 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836596 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596' 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219329: s.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 4 2024-06-11T17:51:18.051 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 90 last_clean_interval [88,89) [v2:127.0.0.1:6874/4165259596,v1:127.0.0.1:6875/4165259596] [v2:127.0.0.1:6876/4165259596,v1:127.0.0.1:6877/4165259596] exists,up 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:51:18.052 INFO:tasks.workunit.client.0.smithi012.stderr:219331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 1 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 2 2024-06-11T17:51:18.053 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 3 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 4 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 5 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 6 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 7 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 8 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: 9' 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836595 2024-06-11T17:51:18.054 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836595 2024-06-11T17:51:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595' 2024-06-11T17:51:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673062 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673062 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062' 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:51:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509536 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509536 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536' 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678037 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678037 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037' 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:51:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182482 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182482 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482' 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925826 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925826 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482 5-403726925826' 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855420 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855420 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482 5-403726925826 6-150323855420' 2024-06-11T17:51:19.345 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.346 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673063 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673063 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063' 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509535 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509535 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535' 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:51:19.347 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678038 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678038 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038' 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182481 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182481 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481' 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925827 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925827 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481 5-403726925827' 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.348 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:51:19.349 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855419 2024-06-11T17:51:19.349 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855419 2024-06-11T17:51:19.349 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481 5-403726925827 6-150323855419' 2024-06-11T17:51:19.349 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.349 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691889 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691889 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481 5-403726925827 6-150323855419 7-171798691889' 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561065 2024-06-11T17:51:19.971 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561065 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481 5-403726925827 6-150323855419 7-171798691889 8-188978561065' 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=408021893123 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 408021893123 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836596 1-42949673063 2-64424509535 3-304942678038 4-107374182481 5-403726925827 6-150323855419 7-171798691889 8-188978561065 9-408021893123' 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836596 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836596 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:19.972 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836596 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.0 seq 21474836596 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836596 -lt 21474836596 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673063 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673063 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673063 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-06-11T17:51:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.1 seq 42949673063 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_pg_stats: seq=171798691888 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691888 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482 5-403726925826 6-150323855420 7-171798691888' 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561064 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561064 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482 5-403726925826 6-150323855420 7-171798691888 8-188978561064' 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:51:19.981 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=408021893122 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 408021893122 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836595 1-42949673062 2-64424509536 3-304942678037 4-107374182482 5-403726925826 6-150323855420 7-171798691888 8-188978561064 9-408021893122' 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836595 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836595 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836595 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836595' 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.0 seq 21474836595 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:51:19.982 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836596 -lt 21474836595 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673062 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673062 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673062 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-06-11T17:51:19.983 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.1 seq 42949673062 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673063 -lt 42949673062 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509536 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509536 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509536 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.2 seq 64424509536 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509536 -lt 64424509536 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678037 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678037 2024-06-11T17:51:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678037 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.3 seq 304942678037 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678038 -lt 304942678037 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182482 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182482 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182482 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-06-11T17:51:21.307 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.4 seq 107374182482 2024-06-11T17:51:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:51:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182482 -lt 107374182482 2024-06-11T17:51:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-403726925826 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cestats: ceph osd last-stat-seq 1 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673063 -lt 42949673063 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509535 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:51:21.326 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509535 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509535 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.2 seq 64424509535 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509536 -lt 64424509535 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678038 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.327 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678038 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678038 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678038' 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.3 seq 304942678038 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678038 -lt 304942678038 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182481 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182481 2024-06-11T17:51:21.328 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182481 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.4 seq 107374182481 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182482 -lt 107374182481 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:21.329 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-403726925827 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-403726925826 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925826 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 403726925826' 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.5 seq 403726925826 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925828 -lt 403726925826 2024-06-11T17:51:22.334 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855420 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855420 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855420 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.6 seq 150323855420 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855421 -lt 150323855420 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691888 2024-06-11T17:51:22.335 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691888 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691888 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.7 seq 171798691888 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691889 -lt 171798691888 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561064 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.336 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:51:22.337 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561064 2024-06-11T17:51:22.337 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.337 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-403726925827 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925827 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 403726925827' 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.5 seq 403726925827 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925828 -lt 403726925827 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855419 2024-06-11T17:51:22.338 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855419 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855419 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.6 seq 150323855419 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855421 -lt 150323855419 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691889 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:51:22.339 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691889 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691889 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.7 seq 171798691889 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691889 -lt 171798691889 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561065 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:22.340 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:51:22.341 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561065 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561065 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561065' 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.8 seq 188978561065 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561065 -lt 188978561065 2024-06-11T17:51:24.125 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-408021893123 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-408021893123 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=408021893123 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: waiting osd.9 seq 408021893123 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 408021893124 -lt 408021893123 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:51:24.126 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:51:24.127 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:ut -d - -f 2 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561064 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561064' 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.8 seq 188978561064 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561065 -lt 188978561064 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:24.128 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-408021893122 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-408021893122 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=408021893122 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: waiting osd.9 seq 408021893122 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 408021893124 -lt 408021893122 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:51:24.129 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:51:24.130 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:219331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 219328 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:51:24.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:51:24.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:51:24.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:51:24.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:51:24.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:51:24.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:24.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:24.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:24.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:24.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:24.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:51:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:51:25.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:25.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:51:25.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:25.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:51:25.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:51:25.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:25.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:25.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:25.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:25.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:25.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:25.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:25.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:51:26.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:51:26.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:26.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:26.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:26.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:26.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:26.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:26.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:26.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:51:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:51:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:27.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:27.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:27.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:27.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:27.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:28.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:28.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:51:29.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:51:29.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:29.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:29.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:29.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:29.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:29.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:51:30.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:51:30.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:30.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:30.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:30.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:30.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:30.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:30.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:50:47.637013+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:30.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:51:31.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:51:31.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:51:31.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:51:31.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:51:31.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:51:31.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:51:31.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:51:25.935089+0000 '>' 2024-06-11T17:50:47.637013+0000 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 222805"' 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 222805' 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:51:32.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/222807: /' 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 222808"' 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 222808' 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 222805 222808' 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 222805 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:51:32.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/222811: /' 2024-06-11T17:51:33.080 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.080 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:33.080 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:33.081 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: _ 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: hinfo_key 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: snapset 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T17:51:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:33.083 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:33.085 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:33.086 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T17:51:33.087 INFO:tasks.workunit.client.0.smithi012.stderr:222807: start osd.5 2024-06-11T17:51:33.277 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.277 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:33.277 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T17:51:33.277 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:33.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: _ 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: hinfo_key 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: snapset 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T17:51:33.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:33.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:51:33.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:33.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:33.282 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:51:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T17:51:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:222811: start osd.9 2024-06-11T17:51:33.283 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T17:51:38.394 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460overy_ops 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2024-06-11T17:51:33.114+0000 7f457e13c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2024-06-11T17:51:33.166+0000 7f457e13c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2024-06-11T17:51:33.170+0000 7f457e13c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:38.395 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 0 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2024-06-11T17:51:33.734+0000 7f457e13c7c0 -1 Falling back to public interface 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 1 2024-06-11T17:51:38.396 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 3 2024-06-11T17:51:38.398 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.398 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:38.398 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2024-06-11T17:51:37.982+0000 7f457e13c7c0 -1 osd.5 97 log_to_monitors true 2024-06-11T17:51:38.404 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:51:38.404 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2024-06-11T17:51:33.306+0000 7f4e3c5657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2024-06-11T17:51:33.342+0000 7f4e3c5657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2024-06-11T17:51:33.342+0000 7f4e3c5657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:38.405 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 0 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2024-06-11T17:51:33.890+0000 7f4e3c5657c0 -1 Falling back to public interface 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 1 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.406 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2024-06-11T17:51:37.338+0000 7f4e3c5657c0 -1 osd.9 97 log_to_monitors true 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:38.407 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 3 2024-06-11T17:51:38.408 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:38.408 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:51:39.869 INFO:tasks.workunit.client.0.smithi012.stderr:222811: osd.9 up in weight 1 up_from 101 up_thru 0 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6874/4105868070,v1:127.0.0.1:6875/4105868070] [v2:127.0.0.1:6876/4105868070,v1:127.0.0.1:6877/4105868070] exists,up 810682a2-ddff-4b14-858b-e45e17b226db 2024-06-11T17:51:39.869 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:51:39.869 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:51:39.870 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 1 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 2 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 3 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 4 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 5 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 6 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 7 2024-06-11T17:51:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 8 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: 9' 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836601 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836601 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601' 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673069 2024-06-11T17:51:39.872 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673069 2024-06-11T17:51:39.873 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069' 2024-06-11T17:51:39.873 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.987 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509542 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509542 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542' 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678044 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678044 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044' 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182487 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182487 2024-06-11T17:51:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044 4-107374182487' 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: Error ENXIO: problem getting command descriptions from osd.5 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855426 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855426 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044 4-107374182487 6-150323855426' 2024-06-11T17:51:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691894 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691894 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044 4-107374182487 6-150323855426 7-171798691894' 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:41.990 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_staeph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 4 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 5 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: osd.5 up in weight 1 up_from 105 up_thru 94 down_at 98 last_clean_interval [94,97) [v2:127.0.0.1:6842/1508536283,v1:127.0.0.1:6843/1508536283] [v2:127.0.0.1:6844/1508536283,v1:127.0.0.1:6845/1508536283] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:51:42.037 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:51:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 1 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 2 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 3 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 4 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 5 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 6 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 7 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 8 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: 9' 2024-06-11T17:51:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:51:43.442 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalonets: seq=188978561070 2024-06-11T17:51:43.442 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561070 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044 4-107374182487 6-150323855426 7-171798691894 8-188978561070' 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696898 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696898 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836601 1-42949673069 2-64424509542 3-304942678044 4-107374182487 6-150323855426 7-171798691894 8-188978561070 9-433791696898' 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836601 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836601 2024-06-11T17:51:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836601 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.0 seq 21474836601 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836602 -lt 21474836601 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673069 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673069 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673069 2024-06-11T17:51:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.1 seq 42949673069 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673069 -lt 42949673069 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509542 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:51:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509542 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836603 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836603 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603' 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673070 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673070 2024-06-11T17:51:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070' 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509543 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509543 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543' 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678045 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678045 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045' 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182489 2024-06-11T17:51:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182489 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489' 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566082 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566082 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489 5-450971566082' 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:51:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=1503238554//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509542 2024-06-11T17:51:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-06-11T17:51:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.2 seq 64424509542 2024-06-11T17:51:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509543 -lt 64424509542 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678044 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678044 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678044 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678044' 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.3 seq 304942678044 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678044 -lt 304942678044 2024-06-11T17:51:44.742 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182487 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182487 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182487 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.4 seq 107374182487 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182487 -lt 107374182487 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855426 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855426 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855426 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-06-11T17:51:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.6 seq 150323855426 2024-06-11T17:51:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/27 2024-06-11T17:51:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855427 2024-06-11T17:51:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489 5-450971566082 6-150323855427' 2024-06-11T17:51:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691896 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691896 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489 5-450971566082 6-150323855427 7-171798691896' 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561071 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561071 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489 5-450971566082 6-150323855427 7-171798691896 8-188978561071' 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696900 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696900 2024-06-11T17:51:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509543 3-304942678045 4-107374182489 5-450971566082 6-150323855427 7-171798691896 8-188978561071 9-433791696900' 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836603 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836603 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836603 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.0 seq 21474836603 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836603 -lt 21474836603 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673070 2024-06-11T17:51:44.886 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:51:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673070 2024-06-11T17:51:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673070 2024-06-11T17:51:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.1 seq 42949673070 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673071 -lt 42949673070 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509543 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509543 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509543 2024-06-11T17:51:45.953 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.2 seq 64424509543 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509544 -lt 64424509543 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-304942678045 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-304942678045 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678045 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 304942678045' 2024-06-11T17:51:45.954 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.3 seq 304942678045 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678046 -lt 304942678045 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182489 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182489 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:45.955 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182489 2024-06-11T17:51:46.098 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtecephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855426 -lt 150323855426 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691894 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691894 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691894 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.7 seq 171798691894 2024-06-11T17:51:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691896 -lt 171798691894 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561070 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561070 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561070 2024-06-11T17:51:46.100 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561070' 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.8 seq 188978561070 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561072 -lt 188978561070 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696898 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696898 2024-06-11T17:51:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696898 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696898' 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: waiting osd.9 seq 433791696898 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696900 -lt 433791696898 2024-06-11T17:51:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:51:47.278 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:51:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:222811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:51:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:51:47.309 INFO:tasks.workunit.client.0.smithi012.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-06-11T17:51:47.309 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.4 seq 107374182489 2024-06-11T17:51:47.309 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:51:47.309 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182489 -lt 107374182489 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-450971566082 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-450971566082 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566082 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 450971566082' 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.5 seq 450971566082 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566082 -lt 450971566082 2024-06-11T17:51:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855427 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855427 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855427 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.6 seq 150323855427 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855428 -lt 150323855427 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691896 2024-06-11T17:51:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691896 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691896 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.7 seq 171798691896 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:51:47.312 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691896 -lt 171798691896 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561071 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561071 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561071 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561071' 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.8 seq 188978561071 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:51:48.752 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561072 -lt 188978561071 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696900 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696900 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696900 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696900' 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: waiting osd.9 seq 433791696900 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696900 -lt 433791696900 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:51:48.753 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:51:48.754 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:222807: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:51:49.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:51:49.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:51:49.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 222808 2024-06-11T17:51:49.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:51:49.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:51:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:51:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:51:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:51:49.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:51:49.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-11T17:51:49.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 225692"' 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 225692' 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 225694"' 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 225694' 2024-06-11T17:51:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 225692 225694' 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/225695: /' 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 225692 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:49.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/225697: /' 2024-06-11T17:51:53.152 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.152 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:53.152 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:51:53.152 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:53.153 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:53.154 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:51:53.155 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:53.155 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:51:53.155 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:53.155 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:53.157 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:53.157 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:53.157 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:51:53.157 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:53.158 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: start osd.3 2024-06-11T17:51:53.159 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:51:53.573 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.573 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:51:53.573 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:51:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T17:51:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:225697: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:51:53.576 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:51:53.578 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:51:53.579 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: start osd.5 2024-06-11T17:51:53.580 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:51:58.027 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2024-06-11T17:51:53.194+0000 7f4773d817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2024-06-11T17:51:53.250+0000 7f4773d817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2024-06-11T17:51:53.250+0000 7f4773d817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:51:58.028 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 0 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2024-06-11T17:51:53.790+0000 7f4773d817c0 -1 Falling back to public interface 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 1 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2024-06-11T17:51:56.886+0000 7f4773d817c0 -1 osd.3 107 log_to_monitors true 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 3 2024-06-11T17:51:58.031 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.031 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225695: osd.3 up in weight 1 up_from 111 up_thru 106 down_at 108 l225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2024-06-11T17:51:53.614+0000 7f9d13f0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.316 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2024-06-11T17:51:53.618+0000 7f9d13f0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2024-06-11T17:51:53.622+0000 7f9d13f0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 0 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:58.317 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2024-06-11T17:51:54.170+0000 7f9d13f0f7c0 -1 Falling back to public interface 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 1 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2 2024-06-11T17:51:58.318 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2024-06-11T17:51:57.506+0000 7f9d13f0f7c0 -1 osd.5 107 log_to_monitors true 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 3 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:51:58.319 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:51:59.567 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [71,107) [v2:127.0.0.1:6826/525111681,v1:127.0.0.1:6827/525111681] [v2:127.0.0.1:6828/525111681,v1:127.0.0.1:6829/525111681] exists,up 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:51:59.568 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 1 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 2 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 3 2024-06-11T17:51:59.569 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 4 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 5 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 6 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 7 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 8 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: 9' 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836607 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836607 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607' 2024-06-11T17:51:59.570 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:51:59.571 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:51:59.571 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673074 2024-06-11T17:51:59.571 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673074 2024-06-11T17:51:59.571 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074' 2024-06-11T17:51:59.571 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:00.907 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/rs.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 4 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: osd.5 up in weight 1 up_from 114 up_thru 105 down_at 108 last_clean_interval [105,107) [v2:127.0.0.1:6842/134461238,v1:127.0.0.1:6843/134461238] [v2:127.0.0.1:6844/134461238,v1:127.0.0.1:6845/134461238] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:52:00.908 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:52:00.909 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 1 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 2 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 3 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 4 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 5 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 6 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 7 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 8 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: 9' 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:52:00.910 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:00.911 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:52:00.911 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836609 2024-06-11T17:52:00.911 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836609 2024-06-11T17:52:00.911 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609' 2024-06-11T17:52:00.911 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225697qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509548 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509548 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548' 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=476741369858 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 476741369858 2024-06-11T17:52:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858' 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182493 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182493 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493' 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=489626271746 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 489626271746 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493 5-489626271746' 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855432 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855432 2024-06-11T17:52:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493 5-489626271746 6-150323855432' 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691900 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691900 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493 5-489626271746 6-150323855432 7-171798691900' 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clo: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673076 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673076 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076' 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509549 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509549 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549' 2024-06-11T17:52:02.626 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=476741369859 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 476741369859 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859' 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182494 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182494 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494' 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:52:02.627 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=489626271747 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 489626271747 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494 5-489626271747' 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855433 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855433 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494 5-489626271747 6-150323855433' 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.628 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flune.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561076 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561076 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493 5-489626271746 6-150323855432 7-171798691900 8-188978561076' 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:52:02.901 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696904 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696904 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673074 2-64424509548 3-476741369858 4-107374182493 5-489626271746 6-150323855432 7-171798691900 8-188978561076 9-433791696904' 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836607 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836607 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836607 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.0 seq 21474836607 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:52:02.902 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836609 -lt 21474836607 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673074 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673074 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673074 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.1 seq 42949673074 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673075 -lt 42949673074 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:02.903 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509548 2024-06-11T17:52:02.904 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:02.904 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:52:03.273 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clonesh_pg_stats: seq=171798691901 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691901 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494 5-489626271747 6-150323855433 7-171798691901' 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561077 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561077 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494 5-489626271747 6-150323855433 7-171798691901 8-188978561077' 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:52:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696906 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696906 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836609 1-42949673076 2-64424509549 3-476741369859 4-107374182494 5-489626271747 6-150323855433 7-171798691901 8-188978561077 9-433791696906' 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836609 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836609 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836609 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836609' 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.0 seq 21474836609 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:52:03.275 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836609 -lt 21474836609 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673076 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673076 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673076 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673076' 2024-06-11T17:52:03.276 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.1 seq 42949673076 2024-06-11T17:52:03.934 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_p.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.934 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509548 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509548 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.2 seq 64424509548 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509549 -lt 64424509548 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-476741369858 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-476741369858 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=476741369858 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-06-11T17:52:03.935 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.3 seq 476741369858 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 476741369860 -lt 476741369858 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182493 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182493 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182493 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.4 seq 107374182493 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:52:03.936 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182494 -lt 107374182493 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-489626271746 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-489626271746 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:03.937 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=489626271746 2024-06-11T17:52:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waig_stats: ceph osd last-stat-seq 1 2024-06-11T17:52:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673076 -lt 42949673076 2024-06-11T17:52:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509549 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509549 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509549 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.2 seq 64424509549 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509549 -lt 64424509549 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-476741369859 2024-06-11T17:52:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-476741369859 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=476741369859 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.3 seq 476741369859 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 476741369860 -lt 476741369859 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182494 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:04.606 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182494 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182494 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.4 seq 107374182494 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182494 -lt 107374182494 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:04.607 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-489626271747 2024-06-11T17:52:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntuting osd.5 seq 489626271746' 2024-06-11T17:52:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.5 seq 489626271746 2024-06-11T17:52:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:52:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 489626271747 -lt 489626271746 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855432 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855432 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855432 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.6 seq 150323855432 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855433 -lt 150323855432 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.302 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691900 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691900 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691900 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.7 seq 171798691900 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691901 -lt 171798691900 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561076 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561076 2024-06-11T17:52:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561076 2024-06-11T17:52:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561076' 2024-06-11T17:52:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.8 seq 188978561076 2024-06-11T17:52:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:52:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561078 -lt 188978561076 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-489626271747 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=489626271747 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 489626271747' 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.5 seq 489626271747 2024-06-11T17:52:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 489626271747 -lt 489626271747 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855433 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855433 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855433 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855433' 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.6 seq 150323855433 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:52:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855434 -lt 150323855433 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691901 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691901 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691901 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.7 seq 171798691901 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691902 -lt 171798691901 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561077 2024-06-11T17:52:05.655 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:05.655 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:52:05.655 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561077 2024-06-11T17:52:06.826 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats:pg_stats: for s in $seqs 2024-06-11T17:52:06.826 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696904 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696904 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696904 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696904' 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: waiting osd.9 seq 433791696904 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696906 -lt 433791696904 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:52:06.828 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:52:06.829 INFO:tasks.workunit.client.0.smithi012.stderr:225695: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:52:06.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:52:06.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:52:06.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 225694 2024-06-11T17:52:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: cut -d - -f 2 2024-06-11T17:52:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561077 2024-06-11T17:52:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561077' 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.8 seq 188978561077 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561078 -lt 188978561077 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696906 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696906 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696906 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696906' 2024-06-11T17:52:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:225697: waiting osd.9 seq 433791696906 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696906 -lt 433791696906 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:52:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:225697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T17:52:07.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:52:07.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T17:52:07.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T17:52:07.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T17:52:07.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:52:07.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T17:52:07.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T17:52:07.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T17:52:07.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T17:52:07.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T17:52:07.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:07.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:07.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:07.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:08.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:08.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T17:52:08.433 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T17:52:08.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:52:08.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:52:08.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:08.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:08.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:08.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:08.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:51:25.935089+0000 '>' 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:08.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:52:09.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:52:09.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:52:09.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:52:09.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:09.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:09.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:09.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:10.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:51:25.935089+0000 '>' 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:10.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:52:11.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:52:11.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:52:11.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:52:11.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:11.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:11.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:11.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:11.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:51:25.935089+0000 '>' 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:11.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:52:12.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:52:12.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:52:12.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:52:12.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:12.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:12.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:12.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:12.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:51:25.935089+0000 '>' 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:12.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T17:52:13.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T17:52:13.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T17:52:13.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T17:52:13.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T17:52:13.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T17:52:13.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T17:52:13.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:52:08.921148+0000 '>' 2024-06-11T17:51:25.935089+0000 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:52:14.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 229122"' 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 229122' 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 229123"' 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 229123' 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 229122 229123' 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 229122 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:52:14.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/229125: /' 2024-06-11T17:52:14.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:52:14.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/229127: /' 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:52:15.337 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:52:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: _ 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: hinfo_key 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: snapset 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:52:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:52:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T17:52:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:229127: start osd.5 2024-06-11T17:52:15.408 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:52:15.409 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: _ 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: hinfo_key 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: snapset 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T17:52:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:52:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:52:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T17:52:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:229125: start osd.3 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2024-06-11T17:52:15.438+0000 7f65ae95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2024-06-11T17:52:15.606+0000 7f65ae95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2024-06-11T17:52:15.606+0000 7f65ae95e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.571 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 0 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2024-06-11T17:52:16.170+0000 7f65ae95e7c0 -1 Falling back to public interface 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.572 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 1 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.573 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.574 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:52:20.574 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 3 2024-06-11T17:52:20.574 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.574 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2024-06-11T17:52:20.570+0000 7f65aovery_ops 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T17:52:20.856 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2024-06-11T17:52:15.374+0000 7fd0a57f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2024-06-11T17:52:15.386+0000 7fd0a57f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2024-06-11T17:52:15.386+0000 7fd0a57f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2024-06-11T17:52:15.946+0000 7fd0a57f67c0 -1 Falling back to public interface 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 0 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.857 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 1 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2 2024-06-11T17:52:20.858 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2024-06-11T17:52:20.034+0000 7fd0a57f67c0 -1 osd.5 115 log_to_monitors true 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 3 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:20.859 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 4 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: osd.5 up in weight 1 up_from 120 up_thru 120 down_at 116 last_clean_interval [114,115) [v2:127.0.0.1:6826/2488293609,v1:127.0.0.1:6827/2488293609] [v2:127.0.0.1:6828/2488293609,v1:127.0.0.1:6829/2488293609] exists,up 4b8ed7b2-643b-4acf-bb5b-0159a0f532eb 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:52:23.841 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:52:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 1 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 2 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 3 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 4 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 5 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 6 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 7 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 8 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: 9' 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:23.843 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:52:23.844 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836614 2024-06-11T17:52:23.844 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836614 2024-06-11T17:52:23.844 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614' 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd e95e7c0 -1 osd.3 115 log_to_monitors true 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 4 2024-06-11T17:52:24.821 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 5 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: osd.3 up in weight 1 up_from 123 up_thru 123 down_at 116 last_clean_interval [111,115) [v2:127.0.0.1:6842/1658368698,v1:127.0.0.1:6843/1658368698] [v2:127.0.0.1:6844/1658368698,v1:127.0.0.1:6845/1658368698] exists,up 8c4184ca-c97f-451d-9abf-0e957cb760ea 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:52:24.822 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:52:24.823 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 1 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 2 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 3 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 4 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 5 2024-06-11T17:52:24.824 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 6 2024-06-11T17:52:24.825 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 7 2024-06-11T17:52:24.825 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 8 2024-06-11T17:52:24.825 INFO:tasks.workunit.client.0.smithi012.stderr:229125: 9' 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standaloin $ids 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673081 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673081 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081' 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509554 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509554 2024-06-11T17:52:25.943 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554' 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=528280977410 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 528280977410 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410' 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182500 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182500 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500' 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.944 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=515396075522 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 515396075522 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500 5-515396075522' 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855439 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855439 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500 5-515396075522 6-150323855439' 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:25.945 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:52:26.395 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperne/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:52:26.395 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836616 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836616 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616' 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673083 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673083 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083' 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509556 2024-06-11T17:52:26.396 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509556 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556' 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=528280977411 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 528280977411 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411' 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182501 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182501 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501' 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:52:26.397 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=515396075524 2024-06-11T17:52:26.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 515396075524 2024-06-11T17:52:26.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501 5-515396075524' 2024-06-11T17:52:26.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:52:26.576 INFO:tasks.workunit.client.0.smithi012.stderr:229125s.sh:2218: flush_pg_stats: seq=171798691907 2024-06-11T17:52:26.576 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691907 2024-06-11T17:52:26.576 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500 5-515396075522 6-150323855439 7-171798691907' 2024-06-11T17:52:26.576 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561083 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561083 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500 5-515396075522 6-150323855439 7-171798691907 8-188978561083' 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696911 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696911 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673081 2-64424509554 3-528280977410 4-107374182500 5-515396075522 6-150323855439 7-171798691907 8-188978561083 9-433791696911' 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836614 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836614 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836614 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.0 seq 21474836614 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836615 -lt 21474836614 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673081 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:52:26.578 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673081 2024-06-11T17:52:26.579 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:26.579 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673081 2024-06-11T17:52:26.579 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673081' 2024-06-11T17:52:26.579 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.1 seq 42949673081 2024-06-11T17:52:27.145 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855440 2024-06-11T17:52:27.145 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855440 2024-06-11T17:52:27.145 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501 5-515396075524 6-150323855440' 2024-06-11T17:52:27.145 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:27.145 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691908 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691908 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501 5-515396075524 6-150323855440 7-171798691908' 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561084 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561084 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501 5-515396075524 6-150323855440 7-171798691908 8-188978561084' 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=433791696912 2024-06-11T17:52:27.146 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 433791696912 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836616 1-42949673083 2-64424509556 3-528280977411 4-107374182501 5-515396075524 6-150323855440 7-171798691908 8-188978561084 9-433791696912' 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836616 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836616 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836616 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836616' 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.0 seq 21474836616 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836616 -lt 21474836616 2024-06-11T17:52:27.147 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.148 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673083 2024-06-11T17:52:27.977 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673082 -lt 42949673081 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509554 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509554 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509554 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509554' 2024-06-11T17:52:27.978 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.2 seq 64424509554 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509556 -lt 64424509554 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-528280977410 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-528280977410 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=528280977410 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 528280977410' 2024-06-11T17:52:27.979 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.3 seq 528280977410 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 528280977411 -lt 528280977410 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182500 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182500 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182500 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182500' 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.4 seq 107374182500 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:52:27.980 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182501 -lt 107374182500 2024-06-11T17:52:27.981 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:27.981 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-515396075522 2024-06-11T17:52:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:229127:ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:52:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673083 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673083 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673083' 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.1 seq 42949673083 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673083 -lt 42949673083 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509556 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509556 2024-06-11T17:52:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509556 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.2 seq 64424509556 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509556 -lt 64424509556 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-528280977411 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-528280977411 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=528280977411 2024-06-11T17:52:28.226 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 528280977411' 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.3 seq 528280977411 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 528280977411 -lt 528280977411 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182501 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182501 2024-06-11T17:52:28.227 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone. //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-515396075522 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=515396075522 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.5 seq 515396075522 2024-06-11T17:52:28.992 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 515396075524 -lt 515396075522 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855439 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855439 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855439 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855439' 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.6 seq 150323855439 2024-06-11T17:52:28.993 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855440 -lt 150323855439 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691907 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691907 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691907 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.7 seq 171798691907 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691908 -lt 171798691907 2024-06-11T17:52:28.994 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:28.995 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561083 2024-06-11T17:52:28.995 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:28.995 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:52:28.995 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561083 2024-06-11T17:52:29.548 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: fclient.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182501 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182501' 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.4 seq 107374182501 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182501 -lt 107374182501 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-515396075524 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-515396075524 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=515396075524 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 515396075524' 2024-06-11T17:52:29.549 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.5 seq 515396075524 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 515396075524 -lt 515396075524 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855440 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855440 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855440 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855440' 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.6 seq 150323855440 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:52:29.550 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855440 -lt 150323855440 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691908 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691908 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691908 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.7 seq 171798691908 2024-06-11T17:52:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clonlush_pg_stats: cut -d - -f 2 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561083 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561083' 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.8 seq 188978561083 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561084 -lt 188978561083 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696911 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:30.825 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696911 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696911 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696911' 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: waiting osd.9 seq 433791696911 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696913 -lt 433791696911 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:52:30.826 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:52:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:229127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:52:30.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:e.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691909 -lt 171798691908 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-188978561084 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-188978561084 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561084 2024-06-11T17:52:31.017 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 188978561084' 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.8 seq 188978561084 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561084 -lt 188978561084 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-433791696912 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-433791696912 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=433791696912 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 433791696912' 2024-06-11T17:52:31.018 INFO:tasks.workunit.client.0.smithi012.stderr:229125: waiting osd.9 seq 433791696912 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 433791696913 -lt 433791696912 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:52:31.019 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:52:31.020 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:52:31.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:52:31.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:52:31.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:52:31.398 INFO:tasks.workunit.client.0.smithi012.stderr:229125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:229125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 229123 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T17:52:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T17:52:31.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T17:52:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T17:52:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:52:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:52:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:52:31.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:52:31.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:52:31.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:52:31.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:52:31.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:52:31.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:52:31.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:52:31.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:52:31.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:52:31.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:52:31.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:52:31.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:52:31.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:52:31.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:52:31.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:52:31.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:52:31.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:52:31.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:52:31.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:52:31.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:52:31.916 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:31.916 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:31.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:52:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:52:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:52:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T17:52:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T17:52:31.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:52:31.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:52:31.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:52:31.922 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T17:52:31.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T17:52:31.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T17:52:31.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T17:52:31.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T17:52:31.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T17:52:31.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:52:31.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T17:52:31.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T17:52:31.926 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T17:52:31.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T17:52:31.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T17:52:31.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T17:52:31.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T17:52:31.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:31.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:31.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T17:52:31.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T17:52:31.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T17:52:31.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T17:52:31.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T17:52:31.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:31.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:31.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T17:52:31.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-11T17:52:31.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T17:52:31.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T17:52:32.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T17:52:32.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:52:32.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:52:32.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:52:32.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:32.467 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:32.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:32.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T17:52:32.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T17:52:32.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T17:52:32.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:52:32.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:52:32.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T17:52:32.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T17:52:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:52:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:52:32.723 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:52:32.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T17:52:32.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T17:52:32.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T17:52:32.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T17:52:32.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T17:52:32.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T17:52:32.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T17:52:32.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T17:52:32.840 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T17:52:32.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T17:52:32.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:33.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:33.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:52:33.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T17:52:33.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-11T17:52:33.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:52:33.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:52:33.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:52:33.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:52:33.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T17:52:33.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:52:33.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b81cb61-6c25-447f-a069-630bfc1c83cc 2024-06-11T17:52:33.959 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 2b81cb61-6c25-447f-a069-630bfc1c83cc 2024-06-11T17:52:33.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2b81cb61-6c25-447f-a069-630bfc1c83cc' 2024-06-11T17:52:33.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:52:33.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhjmhmw9E0OhAAJcPvUXoMoUGPJ/v9AvQjBw== 2024-06-11T17:52:33.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhjmhmw9E0OhAAJcPvUXoMoUGPJ/v9AvQjBw=="}' 2024-06-11T17:52:33.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b81cb61-6c25-447f-a069-630bfc1c83cc -i td/osd-scrub-repair/0/new.json 2024-06-11T17:52:34.263 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:52:34.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T17:52:34.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhjmhmw9E0OhAAJcPvUXoMoUGPJ/v9AvQjBw== --osd-uuid 2b81cb61-6c25-447f-a069-630bfc1c83cc 2024-06-11T17:52:34.320 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.318+0000 7f74a0ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:34.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.346+0000 7f74a0ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:34.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.346+0000 7f74a0ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:34.350 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.350+0000 7f74a0ad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:52:34.350 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.350+0000 7f74a0ad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:52:34.350 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.350+0000 7f74a0ad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T17:52:34.448 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:34.446+0000 7f74a0ad87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T17:52:41.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T17:52:41.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:52:41.837 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T17:52:41.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T17:52:41.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:52:42.247 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T17:52:42.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T17:52:42.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:52:42.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:52:42.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:52:42.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:52:42.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:42.278+0000 7f679f7517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:42.310+0000 7f679f7517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:42.314+0000 7f679f7517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:52:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:42.613 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:52:42.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:52:42.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:42.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:42.872 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:42.870+0000 7f679f7517c0 -1 Falling back to public interface 2024-06-11T17:52:42.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:43.979 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:52:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:52:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:44.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:45.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:45.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:45.313 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:52:45.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:52:45.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:45.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:45.647 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:52:45.646+0000 7f679f7517c0 -1 osd.0 0 log_to_monitors true 2024-06-11T17:52:45.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:46.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:46.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:46.649 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:52:46.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:52:46.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:46.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:46.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:47.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:47.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:47.996 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:52:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:52:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:48.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:49.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:49.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:49.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:52:49.331 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:52:49.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:49.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:49.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:50.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:50.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:50.741 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:52:50.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:52:50.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:50.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:51.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:52.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:52.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:52.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:52:52.078 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:52:52.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:52.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:52.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:53.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:53.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:53.519 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:52:53.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:52:53.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:53.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:54.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:55.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:55.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:55.009 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:52:55.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:52:55.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:55.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:55.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:56.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:56.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:56.343 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:52:56.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:52:56.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:56.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:56.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:52:57.783 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:52:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:58.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:52:59.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:52:59.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:52:59.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:52:59.122 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:52:59.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:52:59.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:52:59.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:00.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:00.569 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:53:00.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:53:00.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:00.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:00.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:01.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:01.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:01.907 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:53:01.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:53:01.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:01.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:02.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:03.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:03.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:03.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:53:03.247 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:53:03.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:03.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:03.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:04.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:04.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:04.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:53:04.640 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:53:04.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:04.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:05.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:06.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:06.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:06.070 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:53:06.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:53:06.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:06.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:07.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:07.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:07.718 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:53:07.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:53:07.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:07.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:08.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:09.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:09.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:09.188 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:53:09.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:53:09.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:09.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:09.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:10.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:10.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:10.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:53:10.863 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:53:10.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:10.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:11.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:12.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:12.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:12.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:53:12.207 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:53:12.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:12.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:12.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:13.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:13.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:13.878 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:53:13.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:53:13.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:13.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:14.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:15.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:15.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:15.381 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:53:15.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:53:15.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:15.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:16.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:16.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:16.721 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T17:53:16.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T17:53:16.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:16.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T17:53:17.053 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1399354920,v1:127.0.0.1:6803/1399354920] [v2:127.0.0.1:6804/1399354920,v1:127.0.0.1:6805/1399354920] exists,up 2b81cb61-6c25-447f-a069-630bfc1c83cc 2024-06-11T17:53:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:53:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:53:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:53:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T17:53:17.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:53:17.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:53:17.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T17:53:17.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:53:17.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58f3c070-4322-4c03-8587-9beadfa286b1 2024-06-11T17:53:17.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 58f3c070-4322-4c03-8587-9beadfa286b1' 2024-06-11T17:53:17.060 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 58f3c070-4322-4c03-8587-9beadfa286b1 2024-06-11T17:53:17.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:53:17.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANj2hmGW+VBBAA6mP9974lPun04wGkJYgFyw== 2024-06-11T17:53:17.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANj2hmGW+VBBAA6mP9974lPun04wGkJYgFyw=="}' 2024-06-11T17:53:17.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58f3c070-4322-4c03-8587-9beadfa286b1 -i td/osd-scrub-repair/1/new.json 2024-06-11T17:53:17.517 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:53:17.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T17:53:17.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANj2hmGW+VBBAA6mP9974lPun04wGkJYgFyw== --osd-uuid 58f3c070-4322-4c03-8587-9beadfa286b1 2024-06-11T17:53:17.574 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.574+0000 7efe9f7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.658+0000 7efe9f7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:17.663 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.662+0000 7efe9f7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:17.663 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.662+0000 7efe9f7fd7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:53:17.663 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.662+0000 7efe9f7fd7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:53:17.663 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.662+0000 7efe9f7fd7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T17:53:17.745 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:17.746+0000 7efe9f7fd7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T17:53:25.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T17:53:25.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:53:25.357 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T17:53:25.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T17:53:25.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:53:25.766 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T17:53:25.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T17:53:25.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:53:25.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:53:25.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:53:25.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:53:25.791 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:25.790+0000 7ff72e36b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:25.793 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:25.790+0000 7ff72e36b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:25.794 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:25.794+0000 7ff72e36b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:53:26.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T17:53:26.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:53:26.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:53:26.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:26.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:26.339 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:26.338+0000 7ff72e36b7c0 -1 Falling back to public interface 2024-06-11T17:53:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:27.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:27.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:27.467 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:53:27.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:53:27.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:27.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:27.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:28.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:28.806 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:53:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:53:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:28.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:29.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:29.322 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:53:29.322+0000 7ff72e36b7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T17:53:30.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:30.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:30.153 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:53:30.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:53:30.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:30.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:30.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:31.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:31.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:31.640 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:53:31.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:53:31.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:31.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:31.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:32.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:32.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:32.976 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:53:32.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:53:32.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:32.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:33.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:34.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:34.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:34.447 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:53:34.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:53:34.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:34.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:34.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:35.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:35.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:35.784 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:53:35.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:53:35.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:35.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:36.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:37.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:37.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:37.136 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:53:37.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:53:37.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:37.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:37.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:38.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:38.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:38.470 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:53:38.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:53:38.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:38.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:38.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:39.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:39.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:39.803 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:53:39.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:53:39.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:39.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:40.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:41.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:41.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:41.145 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:53:41.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:53:41.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:41.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:41.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:42.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:42.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:42.651 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:53:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:53:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:43.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:44.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:44.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:44.059 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:53:44.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:53:44.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:44.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:44.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:45.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:45.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:45.601 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:53:45.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:53:45.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:45.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:45.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:46.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:46.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:46.937 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:53:46.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:53:46.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:46.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:47.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:48.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:48.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:48.275 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:53:48.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:53:48.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:48.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:48.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:49.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:49.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:49.846 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:53:49.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:53:49.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:49.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:50.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:51.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:51.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:51.484 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:53:51.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:53:51.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:51.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:52.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:53.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:53.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:53.037 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:53:53.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:53:53.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:53.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:53.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:54.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:54.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:54.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:53:54.963 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:53:54.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:54.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:55.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:56.966 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:53:56.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:53:56.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:56.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:53:58.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:53:58.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:53:58.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:53:58.444 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:53:58.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:53:58.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:53:59.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:00.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:00.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:00.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:54:00.329 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:54:00.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:00.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T17:54:00.698 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3301004075,v1:127.0.0.1:6811/3301004075] [v2:127.0.0.1:6812/3301004075,v1:127.0.0.1:6813/3301004075] exists,up 58f3c070-4322-4c03-8587-9beadfa286b1 2024-06-11T17:54:00.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:54:00.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:54:00.702 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:54:00.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:54:00.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T17:54:00.705 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:54:00.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2eaca600-2aa5-4f42-ab60-4b32daae9acf 2024-06-11T17:54:00.705 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 2eaca600-2aa5-4f42-ab60-4b32daae9acf 2024-06-11T17:54:00.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2eaca600-2aa5-4f42-ab60-4b32daae9acf' 2024-06-11T17:54:00.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:54:00.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4j2hmLVuQKxAAhSK5VJnN8jtnpmNemv6ZXw== 2024-06-11T17:54:00.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4j2hmLVuQKxAAhSK5VJnN8jtnpmNemv6ZXw=="}' 2024-06-11T17:54:00.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2eaca600-2aa5-4f42-ab60-4b32daae9acf -i td/osd-scrub-repair/2/new.json 2024-06-11T17:54:01.261 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:54:01.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T17:54:01.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4j2hmLVuQKxAAhSK5VJnN8jtnpmNemv6ZXw== --osd-uuid 2eaca600-2aa5-4f42-ab60-4b32daae9acf 2024-06-11T17:54:01.321 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.321+0000 7fb4c44107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:01.388 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.385+0000 7fb4c44107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:01.391 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.389+0000 7fb4c44107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:01.392 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.389+0000 7fb4c44107c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:54:01.392 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.389+0000 7fb4c44107c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:54:01.392 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.389+0000 7fb4c44107c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T17:54:01.490 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:01.489+0000 7fb4c44107c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T17:54:09.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T17:54:09.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:54:09.164 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T17:54:09.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T17:54:09.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:54:09.582 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T17:54:09.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T17:54:09.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:54:09.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:54:09.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:54:09.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:54:09.616 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:09.613+0000 7f6e4f8597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:09.659 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:09.657+0000 7f6e4f8597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:09.661 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:09.657+0000 7f6e4f8597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T17:54:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:54:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T17:54:09.964 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:09.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:10.216 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:10.213+0000 7f6e4f8597c0 -1 Falling back to public interface 2024-06-11T17:54:10.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:11.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:11.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:11.332 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:54:11.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:54:11.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:11.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:11.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:12.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:12.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:12.670 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:54:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:54:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:13.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:13.162 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:13.161+0000 7f6e4f8597c0 -1 osd.2 0 log_to_monitors true 2024-06-11T17:54:14.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:14.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:14.010 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:54:14.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:54:14.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:14.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:14.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:15.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:15.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:15.526 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:54:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:54:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:16.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:16.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:16.940 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:54:16.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:54:16.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:16.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:17.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:18.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:18.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:18.284 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:54:18.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:54:18.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:18.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:18.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:19.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:19.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:19.690 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:54:19.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:54:19.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:19.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:20.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:21.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:21.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:21.024 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:54:21.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:54:21.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:21.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:21.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:22.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:22.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:22.493 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:54:22.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:54:22.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:22.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:23.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:23.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:23.830 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:54:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:54:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:24.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:25.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:25.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:25.179 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:54:25.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:54:25.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:25.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:25.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:26.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:26.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:26.706 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:54:26.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:54:26.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:26.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:27.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:28.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:28.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:28.051 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:54:28.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:54:28.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:28.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:28.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:29.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:29.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:29.476 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:54:29.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:54:29.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:29.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:29.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:30.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:30.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:30.809 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:54:30.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:54:30.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:30.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:31.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:32.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:32.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:32.173 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:54:32.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:54:32.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:32.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:32.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:33.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:33.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:33.940 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:54:33.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:54:33.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:33.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:34.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:35.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:35.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:35.521 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:54:35.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:54:35.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:35.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:36.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:37.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:37.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:37.236 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:54:37.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:54:37.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:37.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:38.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:39.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:39.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:39.284 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:54:39.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:54:39.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:39.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:40.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:41.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:41.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:41.345 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:54:41.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:54:41.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:41.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:41.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:42.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:42.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:42.989 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:54:42.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:54:42.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:42.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:43.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:44.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:44.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:44.404 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:54:44.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:54:44.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:44.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T17:54:44.741 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/106542236,v1:127.0.0.1:6819/106542236] [v2:127.0.0.1:6820/106542236,v1:127.0.0.1:6821/106542236] exists,up 2eaca600-2aa5-4f42-ab60-4b32daae9acf 2024-06-11T17:54:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:54:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:54:44.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:54:44.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:54:44.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T17:54:44.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:54:44.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T17:54:44.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2608169b-5ddc-4b3b-ac20-c2c2b4da8804' 2024-06-11T17:54:44.748 INFO:tasks.workunit.client.0.smithi012.stdout:add osd3 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T17:54:44.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:54:44.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkj2hmVBfuLRAA0wfybjCla6tzl9onjMkAZw== 2024-06-11T17:54:44.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkj2hmVBfuLRAA0wfybjCla6tzl9onjMkAZw=="}' 2024-06-11T17:54:44.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 -i td/osd-scrub-repair/3/new.json 2024-06-11T17:54:45.270 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:54:45.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-11T17:54:45.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkj2hmVBfuLRAA0wfybjCla6tzl9onjMkAZw== --osd-uuid 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T17:54:45.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.329+0000 7f5c5ee577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:45.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.401+0000 7f5c5ee577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:45.407 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.405+0000 7f5c5ee577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:45.407 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.405+0000 7f5c5ee577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:54:45.407 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.405+0000 7f5c5ee577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:54:45.407 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.405+0000 7f5c5ee577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-11T17:54:45.502 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:45.501+0000 7f5c5ee577c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-11T17:54:54.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-11T17:54:54.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:54:54.020 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd3 key to auth repository 2024-06-11T17:54:54.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-11T17:54:54.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:54:54.441 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.3 2024-06-11T17:54:54.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-11T17:54:54.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:54:54.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:54:54.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:54:54.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:54:54.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:54.473+0000 7fc199e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:54.530 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:54.529+0000 7fc199e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:54.531 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:54.529+0000 7fc199e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:54:54.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-11T17:54:54.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:54:54.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T17:54:54.748 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:54:54.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:54:54.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:54:54.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:54.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:54:54.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:54.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:54:55.084 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:55.081+0000 7fc199e247c0 -1 Falling back to public interface 2024-06-11T17:54:55.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:56.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:56.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:56.112 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:54:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:54:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:56.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:54:56.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:57.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:57.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:57.448 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:54:57.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:54:57.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:57.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:54:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:54:57.876 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:54:57.873+0000 7fc199e247c0 -1 osd.3 0 log_to_monitors true 2024-06-11T17:54:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:54:58.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:54:58.792 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:54:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:54:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:54:58.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:54:59.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:00.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:00.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:00.139 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:55:00.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:55:00.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:00.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:00.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:01.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:01.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:01.678 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:55:01.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:55:01.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:01.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:02.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:03.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:03.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:03.181 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:55:03.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:55:03.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:03.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:03.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:04.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:04.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:04.522 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:55:04.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:55:04.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:04.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:04.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:05.922 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:55:05.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:55:05.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:05.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:06.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:07.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:07.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:07.362 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:55:07.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:55:07.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:07.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:07.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:08.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:08.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:08.835 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:55:08.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:55:08.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:08.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:09.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:10.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:10.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:10.427 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:55:10.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:55:10.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:10.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:10.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:11.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:11.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:11.767 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:55:11.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:55:11.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:11.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:12.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:13.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:13.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:13.233 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:55:13.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:55:13.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:13.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:13.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:14.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:14.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:14.570 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:55:14.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:55:14.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:14.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:14.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:15.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:15.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:15.911 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:55:15.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:55:15.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:15.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:16.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:17.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:17.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:17.287 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:55:17.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:55:17.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:17.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:55:19.189 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:55:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:19.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:20.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:20.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:20.633 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:55:20.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:55:20.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:20.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:21.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:22.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:22.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:22.239 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:55:22.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:55:22.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:22.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:22.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:23.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:23.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:23.730 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:55:23.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:55:23.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:23.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:24.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:25.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:25.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:25.581 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:55:25.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:55:25.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:25.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:25.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:26.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:26.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:26.919 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:55:26.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:55:26.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:26.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:27.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:28.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:28.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:28.335 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:55:28.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:55:28.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:28.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:28.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:29.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:29.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:29.707 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T17:55:29.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T17:55:29.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:29.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T17:55:30.092 INFO:tasks.workunit.client.0.smithi012.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1605084707,v1:127.0.0.1:6827/1605084707] [v2:127.0.0.1:6828/1605084707,v1:127.0.0.1:6829/1605084707] exists,up 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T17:55:30.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:55:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:55:30.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:55:30.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4754d516-c288-43a2-806a-b3834ebc59a6 2024-06-11T17:55:30.099 INFO:tasks.workunit.client.0.smithi012.stdout:add osd4 4754d516-c288-43a2-806a-b3834ebc59a6 2024-06-11T17:55:30.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 4754d516-c288-43a2-806a-b3834ebc59a6' 2024-06-11T17:55:30.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:55:30.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSj2hmLaRNBxAAedU9JMqgLhq4FkovUByhqg== 2024-06-11T17:55:30.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSj2hmLaRNBxAAedU9JMqgLhq4FkovUByhqg=="}' 2024-06-11T17:55:30.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4754d516-c288-43a2-806a-b3834ebc59a6 -i td/osd-scrub-repair/4/new.json 2024-06-11T17:55:30.626 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:55:30.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-11T17:55:30.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSj2hmLaRNBxAAedU9JMqgLhq4FkovUByhqg== --osd-uuid 4754d516-c288-43a2-806a-b3834ebc59a6 2024-06-11T17:55:30.688 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.685+0000 7fdf272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:30.762 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.761+0000 7fdf272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:30.764 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.761+0000 7fdf272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:30.765 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.761+0000 7fdf272db7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:55:30.765 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.761+0000 7fdf272db7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:55:30.765 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.761+0000 7fdf272db7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-11T17:55:30.853 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:30.849+0000 7fdf272db7c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-11T17:55:39.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-11T17:55:39.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:55:39.112 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd4 key to auth repository 2024-06-11T17:55:39.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-11T17:55:39.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:55:39.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-11T17:55:39.523 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.4 2024-06-11T17:55:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:55:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:55:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:55:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:55:39.556 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:39.553+0000 7f912f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:39.602 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:39.601+0000 7f912f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:39.604 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:39.601+0000 7f912f74f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:55:39.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:55:39.849 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:55:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:40.164 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:40.161+0000 7f912f74f7c0 -1 Falling back to public interface 2024-06-11T17:55:40.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:41.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:41.214 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:55:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:55:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:41.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:42.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:55:42.553 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:55:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:42.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:42.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:43.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:55:43.025+0000 7f912f74f7c0 -1 osd.4 0 log_to_monitors true 2024-06-11T17:55:43.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:43.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:55:43.890 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:55:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:44.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:45.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:45.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:45.292 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:55:45.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:55:45.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:45.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:45.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:46.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:46.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:46.753 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:55:46.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:55:46.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:46.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:47.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:48.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:48.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:48.087 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:55:48.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:55:48.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:48.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:48.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:49.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:49.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:49.499 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:55:49.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:55:49.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:49.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:49.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:50.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:50.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:50.838 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:55:50.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:55:50.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:50.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:51.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:52.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:52.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:52.219 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:55:52.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:55:52.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:52.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:52.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:53.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:53.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:53.643 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:55:53.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:55:53.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:53.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:53.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:54.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:54.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:54.980 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:55:54.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:55:54.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:54.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:55.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:56.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:56.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:56.315 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:55:56.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:55:56.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:56.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:56.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:57.649 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:55:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:55:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:57.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:55:58.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:55:58.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:55:58.989 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:55:58.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:55:58.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:55:58.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:55:59.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:00.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:00.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:56:00.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:56:00.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:00.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:01.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:01.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:01.671 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:56:01.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:56:01.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:01.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:02.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:03.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:03.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:03.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:56:03.010 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:56:03.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:03.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:03.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:04.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:04.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:04.954 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:56:04.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:56:04.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:04.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:05.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:06.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:06.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:06.973 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:56:06.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:56:06.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:06.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:07.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:08.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:08.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:08.934 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:56:08.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:56:08.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:08.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:09.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:10.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:10.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:10.985 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:56:10.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:56:10.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:10.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:11.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:12.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:12.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:12.693 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:56:12.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:56:12.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:12.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:13.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:14.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:14.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:14.358 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:56:14.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:56:14.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:14.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-11T17:56:14.776 INFO:tasks.workunit.client.0.smithi012.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1516333761,v1:127.0.0.1:6835/1516333761] [v2:127.0.0.1:6836/1516333761,v1:127.0.0.1:6837/1516333761] exists,up 4754d516-c288-43a2-806a-b3834ebc59a6 2024-06-11T17:56:14.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:56:14.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:56:14.780 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:56:14.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T17:56:14.783 INFO:tasks.workunit.client.0.smithi012.stdout:add osd5 b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T17:56:14.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b1f8f08b-bf3f-4f62-b386-9c73197152a2' 2024-06-11T17:56:14.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:56:14.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+j2hmKgQkMBAATcJyhaWmdPNmFrlrEiYY7Q== 2024-06-11T17:56:14.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+j2hmKgQkMBAATcJyhaWmdPNmFrlrEiYY7Q=="}' 2024-06-11T17:56:14.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1f8f08b-bf3f-4f62-b386-9c73197152a2 -i td/osd-scrub-repair/5/new.json 2024-06-11T17:56:15.281 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:56:15.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-11T17:56:15.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+j2hmKgQkMBAATcJyhaWmdPNmFrlrEiYY7Q== --osd-uuid b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T17:56:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.336+0000 7f81775417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:15.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.364+0000 7f81775417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:15.368 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.364+0000 7f81775417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.364+0000 7f81775417c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:56:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.364+0000 7f81775417c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:56:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.364+0000 7f81775417c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-11T17:56:15.540 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:15.536+0000 7f81775417c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-11T17:56:23.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-11T17:56:23.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:56:23.360 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd5 key to auth repository 2024-06-11T17:56:23.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-11T17:56:23.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:56:23.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-11T17:56:23.773 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.5 2024-06-11T17:56:23.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:56:23.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:56:23.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:56:23.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:56:23.805 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:23.804+0000 7fbb9bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:23.840 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:23.836+0000 7fbb9bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:23.842 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:23.840+0000 7fbb9bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:24.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:56:24.142 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:56:24.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:24.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:24.400 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:24.396+0000 7fbb9bde67c0 -1 Falling back to public interface 2024-06-11T17:56:24.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:25.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:25.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:25.511 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:56:25.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:56:25.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:25.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:25.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:26.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:26.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:26.838 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:56:26.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:56:26.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:26.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:27.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:27.320 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:56:27.316+0000 7fbb9bde67c0 -1 osd.5 0 log_to_monitors true 2024-06-11T17:56:28.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:28.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:28.217 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:56:28.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:56:28.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:28.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:28.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:29.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:29.552 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:56:29.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:56:29.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:29.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:30.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:31.098 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:56:31.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:56:31.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:31.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:31.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:32.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:32.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:32.523 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:56:32.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:56:32.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:32.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:32.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:33.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:33.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:33.858 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:56:33.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:56:33.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:33.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:34.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:35.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:35.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:35.195 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:56:35.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:56:35.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:35.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:35.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:36.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:36.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:36.532 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:56:36.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:56:36.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:36.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:36.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:37.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:37.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:37.863 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:56:37.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:56:37.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:37.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:38.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:39.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:39.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:39.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:56:39.199 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:56:39.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:39.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:39.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:40.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:40.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:40.740 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:56:40.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:56:40.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:40.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:42.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:42.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:42.084 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:56:42.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:56:42.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:42.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:42.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:43.445 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:56:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:56:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:43.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:44.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:44.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:44.782 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:56:44.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:56:44.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:44.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:45.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:46.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:46.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:46.119 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:56:46.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:56:46.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:46.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:46.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:47.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:47.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:47.926 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:56:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:56:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:48.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:49.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:49.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:49.543 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:56:49.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:56:49.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:49.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:50.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:51.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:51.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:51.534 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:56:51.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:56:51.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:51.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:52.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:53.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:53.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:53.313 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:56:53.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:56:53.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:53.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:53.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:54.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:54.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:54.973 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T17:56:54.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T17:56:54.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:54.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:55.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:56.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:56.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:56.650 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T17:56:56.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T17:56:56.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:56.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:57.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:58.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:58.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:58.286 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T17:56:58.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T17:56:58.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:58.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:56:58.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:56:59.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:56:59.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:56:59.783 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T17:56:59.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T17:56:59.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:56:59.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T17:57:00.122 INFO:tasks.workunit.client.0.smithi012.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3001490129,v1:127.0.0.1:6843/3001490129] [v2:127.0.0.1:6844/3001490129,v1:127.0.0.1:6845/3001490129] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T17:57:00.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:57:00.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:57:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:57:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:57:00.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-11T17:57:00.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:57:00.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b67f504-09ec-422b-9276-d4e6a3f540df 2024-06-11T17:57:00.129 INFO:tasks.workunit.client.0.smithi012.stdout:add osd6 7b67f504-09ec-422b-9276-d4e6a3f540df 2024-06-11T17:57:00.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7b67f504-09ec-422b-9276-d4e6a3f540df' 2024-06-11T17:57:00.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:57:00.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsj2hmS+6yCBAAS8mi8H/0A2rYwPi05lV8BA== 2024-06-11T17:57:00.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsj2hmS+6yCBAAS8mi8H/0A2rYwPi05lV8BA=="}' 2024-06-11T17:57:00.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b67f504-09ec-422b-9276-d4e6a3f540df -i td/osd-scrub-repair/6/new.json 2024-06-11T17:57:00.656 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:57:00.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-11T17:57:00.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsj2hmS+6yCBAAS8mi8H/0A2rYwPi05lV8BA== --osd-uuid 7b67f504-09ec-422b-9276-d4e6a3f540df 2024-06-11T17:57:00.718 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.716+0000 7f7c2e7397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:00.783 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.780+0000 7f7c2e7397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:00.785 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.780+0000 7f7c2e7397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:00.785 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.784+0000 7f7c2e7397c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:57:00.785 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.784+0000 7f7c2e7397c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:57:00.786 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.784+0000 7f7c2e7397c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-11T17:57:00.862 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:00.860+0000 7f7c2e7397c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-11T17:57:06.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-11T17:57:06.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:57:06.999 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd6 key to auth repository 2024-06-11T17:57:06.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-11T17:57:06.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:57:07.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-11T17:57:07.455 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.6 2024-06-11T17:57:07.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:57:07.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:57:07.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:57:07.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:57:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:07.488+0000 7f53469aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:07.518 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:07.516+0000 7f53469aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:07.519 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:07.516+0000 7f53469aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:57:07.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:07.812 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:57:07.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:57:07.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:07.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:08.076 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:08.072+0000 7f53469aa7c0 -1 Falling back to public interface 2024-06-11T17:57:08.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:09.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:09.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:09.144 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:57:09.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:57:09.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:09.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:09.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:10.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:10.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:10.472 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:57:10.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:57:10.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:10.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:10.578 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:10.576+0000 7f53469aa7c0 -1 osd.6 0 log_to_monitors true 2024-06-11T17:57:10.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:11.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:11.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:11.872 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:57:11.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:57:11.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:11.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:12.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:13.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:13.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:13.234 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:57:13.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:57:13.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:13.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:13.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:14.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:14.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:14.575 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:57:14.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:57:14.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:14.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:14.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:15.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:15.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:15.915 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:57:15.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:57:15.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:15.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:16.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:17.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:17.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:17.252 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:57:17.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:57:17.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:17.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:17.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:18.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:18.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:18.827 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:57:18.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:57:18.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:18.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:19.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:20.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:20.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:20.251 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:57:20.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:57:20.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:20.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:20.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:21.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:21.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:21.648 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:57:21.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:57:21.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:21.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:22.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:23.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:23.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:23.059 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:57:23.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:57:23.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:23.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:23.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:24.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:24.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:24.474 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:57:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:57:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:24.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:25.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:25.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:25.875 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:57:25.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:57:25.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:25.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:26.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:27.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:27.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:27.212 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:57:27.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:57:27.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:27.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:28.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:29.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:29.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:29.021 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:57:29.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:57:29.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:29.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:30.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:31.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:31.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:31.032 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:57:31.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:57:31.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:31.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:31.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:32.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:32.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:32.901 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:57:32.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:57:32.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:32.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:33.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:34.428 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:57:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:57:34.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:34.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:34.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:35.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:35.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:35.953 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:57:35.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:57:35.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:35.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:36.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:37.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:37.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:37.623 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:57:37.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:57:37.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:37.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-11T17:57:38.064 INFO:tasks.workunit.client.0.smithi012.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/787830326,v1:127.0.0.1:6851/787830326] [v2:127.0.0.1:6852/787830326,v1:127.0.0.1:6853/787830326] exists,up 7b67f504-09ec-422b-9276-d4e6a3f540df 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:57:38.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:57:38.068 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:57:38.069 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:57:38.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:57:38.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c620066f-0706-4653-8262-db902bbb02db 2024-06-11T17:57:38.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 c620066f-0706-4653-8262-db902bbb02db' 2024-06-11T17:57:38.072 INFO:tasks.workunit.client.0.smithi012.stdout:add osd7 c620066f-0706-4653-8262-db902bbb02db 2024-06-11T17:57:38.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:57:38.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASkGhmdSlwBRAAICdCQNBNjYAmE2yvU1Ud2g== 2024-06-11T17:57:38.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASkGhmdSlwBRAAICdCQNBNjYAmE2yvU1Ud2g=="}' 2024-06-11T17:57:38.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c620066f-0706-4653-8262-db902bbb02db -i td/osd-scrub-repair/7/new.json 2024-06-11T17:57:38.492 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:57:38.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-11T17:57:38.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASkGhmdSlwBRAAICdCQNBNjYAmE2yvU1Ud2g== --osd-uuid c620066f-0706-4653-8262-db902bbb02db 2024-06-11T17:57:38.552 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.548+0000 7f1536d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:38.560 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.556+0000 7f1536d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:38.562 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.556+0000 7f1536d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:38.562 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.556+0000 7f1536d407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:57:38.562 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.560+0000 7f1536d407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:57:38.562 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.560+0000 7f1536d407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-11T17:57:38.685 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:38.680+0000 7f1536d407c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-11T17:57:45.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-11T17:57:45.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:57:45.014 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd7 key to auth repository 2024-06-11T17:57:45.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-11T17:57:45.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:57:45.435 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.7 2024-06-11T17:57:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-11T17:57:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:57:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:57:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:57:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:57:45.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:45.464+0000 7fced140f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:45.506 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:45.504+0000 7fced140f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:45.507 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:45.504+0000 7fced140f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:57:45.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-11T17:57:45.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:57:45.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-11T17:57:45.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:57:45.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:57:45.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:45.809 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:57:45.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:57:45.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:45.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:46.068 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:46.064+0000 7fced140f7c0 -1 Falling back to public interface 2024-06-11T17:57:46.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:47.184 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:57:47.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:57:47.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:47.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:47.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:48.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:48.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:48.518 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:57:48.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:57:48.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:48.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:48.713 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:57:48.708+0000 7fced140f7c0 -1 osd.7 0 log_to_monitors true 2024-06-11T17:57:48.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:49.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:49.857 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:49.857 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:57:49.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:57:49.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:49.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:50.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:51.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:51.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:51.195 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:57:51.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:57:51.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:51.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:51.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:52.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:52.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:52.608 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:57:52.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:57:52.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:52.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:52.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:53.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:53.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:53.952 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:57:53.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:57:53.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:53.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:54.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:55.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:55.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:55.284 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:57:55.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:57:55.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:55.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:55.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:56.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:56.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:56.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:57:56.710 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:57:56.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:56.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:57.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:58.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:58.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:58.065 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:57:58.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:57:58.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:58.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:58.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:57:59.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:57:59.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:57:59.396 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:57:59.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:57:59.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:57:59.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:57:59.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:00.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:00.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:00.791 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:58:00.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:58:00.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:00.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:01.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:02.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:02.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:02.216 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:58:02.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:58:02.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:02.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:02.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:03.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:03.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:03.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:58:03.650 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:58:03.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:03.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:04.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:05.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:05.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:05.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:58:05.045 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:58:05.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:05.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:05.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:06.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:06.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:06.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:58:06.943 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:58:06.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:06.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:07.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:08.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:08.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:08.424 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:58:08.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:58:08.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:08.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:09.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:10.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:10.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:10.203 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:58:10.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:58:10.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:10.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:10.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:11.824 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:58:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:58:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:12.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:13.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:13.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:13.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:58:13.350 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:58:13.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:13.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:14.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:14.710 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:58:14.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:58:14.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:14.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-11T17:58:15.082 INFO:tasks.workunit.client.0.smithi012.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3799886937,v1:127.0.0.1:6859/3799886937] [v2:127.0.0.1:6860/3799886937,v1:127.0.0.1:6861/3799886937] exists,up c620066f-0706-4653-8262-db902bbb02db 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:58:15.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:58:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:58:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:58:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:58:15.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:58:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:58:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:58:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:58:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-11T17:58:15.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:58:15.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f749775d-8f5e-4fb0-9afe-7ecaf32467b9 2024-06-11T17:58:15.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 f749775d-8f5e-4fb0-9afe-7ecaf32467b9' 2024-06-11T17:58:15.089 INFO:tasks.workunit.client.0.smithi012.stdout:add osd8 f749775d-8f5e-4fb0-9afe-7ecaf32467b9 2024-06-11T17:58:15.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:58:15.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3kGhmyx6gBhAAS7Uo7NIM07ao9UVKFDwfAQ== 2024-06-11T17:58:15.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3kGhmyx6gBhAAS7Uo7NIM07ao9UVKFDwfAQ=="}' 2024-06-11T17:58:15.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f749775d-8f5e-4fb0-9afe-7ecaf32467b9 -i td/osd-scrub-repair/8/new.json 2024-06-11T17:58:15.600 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:58:15.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-11T17:58:15.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3kGhmyx6gBhAAS7Uo7NIM07ao9UVKFDwfAQ== --osd-uuid f749775d-8f5e-4fb0-9afe-7ecaf32467b9 2024-06-11T17:58:15.661 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.655+0000 7fa24c4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:15.727 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.723+0000 7fa24c4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:15.730 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.723+0000 7fa24c4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.727+0000 7fa24c4477c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:58:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.727+0000 7fa24c4477c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:58:15.731 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.727+0000 7fa24c4477c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-11T17:58:15.828 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:15.827+0000 7fa24c4477c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-11T17:58:22.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-11T17:58:22.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:58:22.494 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd8 key to auth repository 2024-06-11T17:58:22.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-11T17:58:22.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:58:22.911 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.8 2024-06-11T17:58:22.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-11T17:58:22.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:58:22.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:58:22.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:58:22.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:58:22.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:22.939+0000 7ff4d6b907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:22.999+0000 7ff4d6b907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:23.006 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:23.003+0000 7ff4d6b907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:23.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-11T17:58:23.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:58:23.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-11T17:58:23.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:58:23.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:58:23.247 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:58:23.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:23.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:58:23.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:23.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:23.540 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:23.535+0000 7ff4d6b907c0 -1 Falling back to public interface 2024-06-11T17:58:23.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:24.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:24.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:24.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:58:24.572 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:58:24.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:24.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:24.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:25.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:25.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:25.899 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:58:25.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:58:25.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:25.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:26.039 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:26.035+0000 7ff4d6b907c0 -1 osd.8 0 log_to_monitors true 2024-06-11T17:58:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:27.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:27.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:27.345 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:58:27.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:58:27.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:27.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:27.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:28.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:28.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:28.884 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:58:28.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:58:28.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:28.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:29.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:30.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:30.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:30.237 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:58:30.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:58:30.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:30.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:30.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:31.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:31.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:31.573 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:58:31.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:58:31.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:31.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:32.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:33.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:33.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:33.134 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:58:33.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:58:33.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:33.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:33.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:34.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:34.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:34.603 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:58:34.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:58:34.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:34.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:34.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:35.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:35.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:35.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:58:35.947 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:58:35.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:35.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:36.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:37.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:37.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:37.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:58:37.282 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:58:37.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:37.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:37.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:38.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:38.620 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:58:38.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:58:38.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:39.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:40.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:40.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:40.139 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:58:40.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:58:40.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:40.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:40.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:41.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:41.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:41.484 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:58:41.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:58:41.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:41.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:41.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:42.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:42.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:42.939 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:58:42.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:58:42.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:42.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:43.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:44.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:44.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:44.721 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:58:44.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:58:44.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:44.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:45.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:46.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:46.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:46.828 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:58:46.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:58:46.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:46.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:47.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:48.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:48.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:48.309 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:58:48.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:58:48.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:48.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:48.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:49.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:49.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:49.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:58:49.679 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:58:49.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:49.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:50.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:58:51.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:58:51.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:58:51.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:58:51.023 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:58:51.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:58:51.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-11T17:58:51.471 INFO:tasks.workunit.client.0.smithi012.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3284213239,v1:127.0.0.1:6867/3284213239] [v2:127.0.0.1:6868/3284213239,v1:127.0.0.1:6869/3284213239] exists,up f749775d-8f5e-4fb0-9afe-7ecaf32467b9 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-11T17:58:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T17:58:51.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T17:58:51.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T17:58:51.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T17:58:51.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T17:58:51.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T17:58:51.480 INFO:tasks.workunit.client.0.smithi012.stdout:add osd9 b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T17:58:51.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508' 2024-06-11T17:58:51.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T17:58:51.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbkGhmB43bHRAApLKy1bmrR2gG94V8VGHRsQ== 2024-06-11T17:58:51.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbkGhmB43bHRAApLKy1bmrR2gG94V8VGHRsQ=="}' 2024-06-11T17:58:51.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 -i td/osd-scrub-repair/9/new.json 2024-06-11T17:58:52.012 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:58:52.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-11T17:58:52.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbkGhmB43bHRAApLKy1bmrR2gG94V8VGHRsQ== --osd-uuid b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T17:58:52.073 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.067+0000 7f0e4d7837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:52.209 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.203+0000 7f0e4d7837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:52.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.207+0000 7f0e4d7837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:52.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.207+0000 7f0e4d7837c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:58:52.212 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.207+0000 7f0e4d7837c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:58:52.212 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.207+0000 7f0e4d7837c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-11T17:58:52.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:52.343+0000 7f0e4d7837c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-11T17:58:59.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-11T17:58:59.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T17:58:59.432 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd9 key to auth repository 2024-06-11T17:58:59.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-11T17:58:59.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T17:58:59.848 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.9 2024-06-11T17:58:59.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-11T17:58:59.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T17:58:59.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T17:58:59.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T17:58:59.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T17:58:59.882 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:59.879+0000 7fb011b537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:59.923 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:59.919+0000 7fb011b537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:58:59.925 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:58:59.919+0000 7fb011b537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T17:59:00.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T17:59:00.194 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T17:59:00.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:00.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:00.467 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:59:00.463+0000 7fb011b537c0 -1 Falling back to public interface 2024-06-11T17:59:00.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:01.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:01.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:01.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T17:59:01.517 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T17:59:01.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:01.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:01.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:02.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:02.849 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T17:59:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T17:59:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:02.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:03.117 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T17:59:03.111+0000 7fb011b537c0 -1 osd.9 0 log_to_monitors true 2024-06-11T17:59:03.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:04.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:04.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:04.347 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T17:59:04.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T17:59:04.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:04.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:04.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:05.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:05.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:05.687 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T17:59:05.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T17:59:05.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:05.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:06.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:07.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:07.203 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T17:59:07.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T17:59:07.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:07.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:07.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:08.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:08.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:08.551 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T17:59:08.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T17:59:08.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:08.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:08.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:09.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:09.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:09.890 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T17:59:09.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T17:59:09.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:09.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:10.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:11.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:11.287 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T17:59:11.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T17:59:11.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:11.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:11.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:12.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:12.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:12.742 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T17:59:12.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T17:59:12.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:12.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:13.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:14.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:14.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:14.226 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T17:59:14.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T17:59:14.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:14.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:14.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:15.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:15.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:15.567 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T17:59:15.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T17:59:15.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:15.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:16.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:17.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:17.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:17.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T17:59:17.055 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T17:59:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:17.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:17.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:18.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:18.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:18.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T17:59:18.425 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T17:59:18.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:18.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:20.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:20.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:20.191 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T17:59:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T17:59:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:20.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:21.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:22.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:22.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:22.302 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T17:59:22.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T17:59:22.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:22.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:22.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:23.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:23.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:23.718 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T17:59:23.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T17:59:23.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:23.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:24.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:25.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:25.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:25.673 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T17:59:25.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T17:59:25.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:25.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:26.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:27.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:27.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:27.647 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T17:59:27.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T17:59:27.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:27.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:28.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:29.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:29.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:29.226 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T17:59:29.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T17:59:29.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:29.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:29.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T17:59:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T17:59:30.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T17:59:30.745 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T17:59:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T17:59:30.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T17:59:30.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T17:59:31.083 INFO:tasks.workunit.client.0.smithi012.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1432683170,v1:127.0.0.1:6875/1432683170] [v2:127.0.0.1:6876/1432683170,v1:127.0.0.1:6877/1432683170] exists,up b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T17:59:31.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T17:59:31.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T17:59:31.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T17:59:31.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-11T17:59:31.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T17:59:31.404 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T17:59:31.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T17:59:31.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T17:59:31.870 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T17:59:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:59:32.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T17:59:37.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-11T17:59:37.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:59:37.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:59:37.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:59:37.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:59:37.262 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:59:37.262 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:59:37.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:59:37.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:59:37.263 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:59:37.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:59:37.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:59:37.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:59:37.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:59:37.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:59:37.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:59:37.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:59:37.896 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:59:37.896 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:59:37.896 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:37.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:59:38.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836561 2024-06-11T17:59:38.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836561 2024-06-11T17:59:38.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561' 2024-06-11T17:59:38.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:59:38.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673039 2024-06-11T17:59:38.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673039 2024-06-11T17:59:38.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039' 2024-06-11T17:59:38.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:59:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509510 2024-06-11T17:59:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509510 2024-06-11T17:59:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510' 2024-06-11T17:59:38.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:59:38.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345975 2024-06-11T17:59:38.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345975 2024-06-11T17:59:38.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975' 2024-06-11T17:59:38.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.568 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:59:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182448 2024-06-11T17:59:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182448 2024-06-11T17:59:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448' 2024-06-11T17:59:38.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:59:38.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018920 2024-06-11T17:59:38.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018920 2024-06-11T17:59:38.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448 5-128849018920' 2024-06-11T17:59:38.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:38.861 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:59:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855391 2024-06-11T17:59:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855391 2024-06-11T17:59:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448 5-128849018920 6-150323855391' 2024-06-11T17:59:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:39.012 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:59:39.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691866 2024-06-11T17:59:39.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691866 2024-06-11T17:59:39.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448 5-128849018920 6-150323855391 7-171798691866' 2024-06-11T17:59:39.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:39.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:59:39.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528333 2024-06-11T17:59:39.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528333 2024-06-11T17:59:39.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448 5-128849018920 6-150323855391 7-171798691866 8-193273528333' 2024-06-11T17:59:39.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:39.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:59:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364807 2024-06-11T17:59:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364807 2024-06-11T17:59:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509510 3-85899345975 4-107374182448 5-128849018920 6-150323855391 7-171798691866 8-193273528333 9-214748364807' 2024-06-11T17:59:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:39.454 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:39.454 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836561 2024-06-11T17:59:39.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:59:39.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836561 2024-06-11T17:59:39.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:39.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836561 2024-06-11T17:59:39.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-06-11T17:59:39.458 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836561 2024-06-11T17:59:39.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:59:39.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836561 -lt 21474836561 2024-06-11T17:59:39.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:39.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:39.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673039 2024-06-11T17:59:39.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:59:39.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673039 2024-06-11T17:59:39.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:39.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673039 2024-06-11T17:59:39.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-06-11T17:59:39.789 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949673039 2024-06-11T17:59:39.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:59:40.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673039 -lt 42949673039 2024-06-11T17:59:40.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:40.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:40.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509510 2024-06-11T17:59:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:59:40.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509510 2024-06-11T17:59:40.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:40.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509510 2024-06-11T17:59:40.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-06-11T17:59:40.148 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509510 2024-06-11T17:59:40.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:59:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509510 -lt 64424509510 2024-06-11T17:59:40.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:40.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345975 2024-06-11T17:59:40.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:40.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:59:40.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345975 2024-06-11T17:59:40.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:40.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345975 2024-06-11T17:59:40.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345975' 2024-06-11T17:59:40.500 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345975 2024-06-11T17:59:40.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:59:40.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345975 -lt 85899345975 2024-06-11T17:59:40.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:40.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:40.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182448 2024-06-11T17:59:40.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:59:40.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182448 2024-06-11T17:59:40.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:40.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182448 2024-06-11T17:59:40.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-06-11T17:59:40.849 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.4 seq 107374182448 2024-06-11T17:59:40.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:59:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182449 -lt 107374182448 2024-06-11T17:59:41.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:41.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018920 2024-06-11T17:59:41.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:41.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:59:41.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018920 2024-06-11T17:59:41.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:41.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018920 2024-06-11T17:59:41.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-06-11T17:59:41.235 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.5 seq 128849018920 2024-06-11T17:59:41.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:59:41.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018921 -lt 128849018920 2024-06-11T17:59:41.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:41.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855391 2024-06-11T17:59:41.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:41.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:59:41.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855391 2024-06-11T17:59:41.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:41.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855391 2024-06-11T17:59:41.552 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.6 seq 150323855391 2024-06-11T17:59:41.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-06-11T17:59:41.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:59:41.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855391 -lt 150323855391 2024-06-11T17:59:41.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:41.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691866 2024-06-11T17:59:41.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:59:41.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:41.915 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691866 2024-06-11T17:59:41.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691866 2024-06-11T17:59:41.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-06-11T17:59:41.917 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.7 seq 171798691866 2024-06-11T17:59:41.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:59:42.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691867 -lt 171798691866 2024-06-11T17:59:42.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:42.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528333 2024-06-11T17:59:42.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:42.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:59:42.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528333 2024-06-11T17:59:42.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:42.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528333 2024-06-11T17:59:42.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-06-11T17:59:42.238 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.8 seq 193273528333 2024-06-11T17:59:42.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:59:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528333 -lt 193273528333 2024-06-11T17:59:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:42.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-214748364807 2024-06-11T17:59:42.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:42.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:59:42.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:42.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-214748364807 2024-06-11T17:59:42.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364807 2024-06-11T17:59:42.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 214748364807' 2024-06-11T17:59:42.684 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.9 seq 214748364807 2024-06-11T17:59:42.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:59:43.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364808 -lt 214748364807 2024-06-11T17:59:43.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:59:43.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:59:43.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:59:43.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T17:59:43.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:59:43.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:59:43.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:59:43.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:59:43.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:59:43.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:59:43.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:59:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T17:59:43.791 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:59:43.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:59:43.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:59:44.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T17:59:44.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:59:44.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:59:44.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-06-11T17:59:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T17:59:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T17:59:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-06-11T17:59:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T17:59:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-11T17:59:44.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T17:59:44.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T17:59:45.468 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T17:59:45.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T17:59:46.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-06-11T17:59:46.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-11T17:59:47.343 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 2 allow_ec_overwrites to true 2024-06-11T17:59:47.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T17:59:47.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T17:59:47.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T17:59:47.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T17:59:47.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T17:59:47.369 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T17:59:47.369 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T17:59:47.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T17:59:47.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T17:59:47.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T17:59:47.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T17:59:47.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T17:59:47.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:47.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T17:59:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836564 2024-06-11T17:59:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836564 2024-06-11T17:59:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564' 2024-06-11T17:59:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:48.016 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T17:59:48.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673042 2024-06-11T17:59:48.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673042 2024-06-11T17:59:48.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042' 2024-06-11T17:59:48.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:48.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T17:59:48.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509513 2024-06-11T17:59:48.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509513 2024-06-11T17:59:48.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513' 2024-06-11T17:59:48.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:48.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T17:59:48.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345978 2024-06-11T17:59:48.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345978 2024-06-11T17:59:48.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978' 2024-06-11T17:59:48.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:48.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T17:59:48.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182451 2024-06-11T17:59:48.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182451 2024-06-11T17:59:48.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451' 2024-06-11T17:59:48.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:48.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T17:59:49.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018924 2024-06-11T17:59:49.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018924 2024-06-11T17:59:49.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451 5-128849018924' 2024-06-11T17:59:49.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:49.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T17:59:49.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855394 2024-06-11T17:59:49.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855394 2024-06-11T17:59:49.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451 5-128849018924 6-150323855394' 2024-06-11T17:59:49.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:49.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T17:59:49.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691869 2024-06-11T17:59:49.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691869 2024-06-11T17:59:49.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451 5-128849018924 6-150323855394 7-171798691869' 2024-06-11T17:59:49.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:49.452 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T17:59:49.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528336 2024-06-11T17:59:49.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528336 2024-06-11T17:59:49.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451 5-128849018924 6-150323855394 7-171798691869 8-193273528336' 2024-06-11T17:59:49.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T17:59:49.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T17:59:49.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364810 2024-06-11T17:59:49.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364810 2024-06-11T17:59:49.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509513 3-85899345978 4-107374182451 5-128849018924 6-150323855394 7-171798691869 8-193273528336 9-214748364810' 2024-06-11T17:59:49.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:49.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836564 2024-06-11T17:59:49.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:49.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T17:59:49.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836564 2024-06-11T17:59:49.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:49.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836564 2024-06-11T17:59:49.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-06-11T17:59:49.748 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836564 2024-06-11T17:59:49.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T17:59:50.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836564 -lt 21474836564 2024-06-11T17:59:50.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:50.067 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673042 2024-06-11T17:59:50.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:50.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T17:59:50.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:50.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673042 2024-06-11T17:59:50.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673042 2024-06-11T17:59:50.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-06-11T17:59:50.072 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949673042 2024-06-11T17:59:50.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T17:59:50.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673042 -lt 42949673042 2024-06-11T17:59:50.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:50.404 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509513 2024-06-11T17:59:50.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:50.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T17:59:50.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509513 2024-06-11T17:59:50.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:50.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509513 2024-06-11T17:59:50.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-06-11T17:59:50.409 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509513 2024-06-11T17:59:50.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T17:59:50.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509513 -lt 64424509513 2024-06-11T17:59:50.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:50.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345978 2024-06-11T17:59:50.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:50.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T17:59:50.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345978 2024-06-11T17:59:50.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:50.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345978 2024-06-11T17:59:50.741 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.3 seq 85899345978 2024-06-11T17:59:50.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-06-11T17:59:50.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T17:59:51.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345979 -lt 85899345978 2024-06-11T17:59:51.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:51.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:51.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182451 2024-06-11T17:59:51.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T17:59:51.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182451 2024-06-11T17:59:51.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:51.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182451 2024-06-11T17:59:51.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-06-11T17:59:51.081 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.4 seq 107374182451 2024-06-11T17:59:51.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T17:59:51.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182452 -lt 107374182451 2024-06-11T17:59:51.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:51.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018924 2024-06-11T17:59:51.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:51.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T17:59:51.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018924 2024-06-11T17:59:51.405 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:51.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018924 2024-06-11T17:59:51.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-06-11T17:59:51.407 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.5 seq 128849018924 2024-06-11T17:59:51.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T17:59:51.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018924 -lt 128849018924 2024-06-11T17:59:51.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:51.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855394 2024-06-11T17:59:51.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:51.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T17:59:51.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855394 2024-06-11T17:59:51.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:51.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855394 2024-06-11T17:59:51.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-06-11T17:59:51.743 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.6 seq 150323855394 2024-06-11T17:59:51.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T17:59:52.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855394 -lt 150323855394 2024-06-11T17:59:52.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:52.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691869 2024-06-11T17:59:52.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:52.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T17:59:52.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691869 2024-06-11T17:59:52.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:52.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691869 2024-06-11T17:59:52.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-06-11T17:59:52.291 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.7 seq 171798691869 2024-06-11T17:59:52.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T17:59:52.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691870 -lt 171798691869 2024-06-11T17:59:52.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:52.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528336 2024-06-11T17:59:52.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:52.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T17:59:52.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528336 2024-06-11T17:59:52.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:52.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528336 2024-06-11T17:59:52.641 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.8 seq 193273528336 2024-06-11T17:59:52.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-06-11T17:59:52.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T17:59:52.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528336 -lt 193273528336 2024-06-11T17:59:52.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T17:59:52.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T17:59:52.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-214748364810 2024-06-11T17:59:52.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T17:59:52.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T17:59:52.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-214748364810 2024-06-11T17:59:52.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364810 2024-06-11T17:59:52.983 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.9 seq 214748364810 2024-06-11T17:59:52.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-06-11T17:59:52.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T17:59:53.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364811 -lt 214748364810 2024-06-11T17:59:53.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T17:59:53.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:59:53.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:59:53.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T17:59:53.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T17:59:53.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T17:59:53.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T17:59:53.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T17:59:53.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T17:59:53.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T17:59:53.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T17:59:54.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T17:59:54.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T17:59:54.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T17:59:54.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T17:59:54.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T17:59:54.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T17:59:54.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T17:59:54.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T17:59:54.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T17:59:54.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T17:59:54.910 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T17:59:54.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T17:59:55.413 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T17:59:55.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T17:59:55.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T17:59:55.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T17:59:56.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-11T17:59:56.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-11T17:59:56.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T17:59:56.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:59:56.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T17:59:56.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-11T17:59:56.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-11T17:59:56.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-11T17:59:56.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T17:59:56.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-11T17:59:56.546 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-11T17:59:56.546 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:9 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr:7' 2024-06-11T17:59:56.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-11T17:59:56.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-11T17:59:56.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-11T17:59:56.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T17:59:56.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T17:59:56.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T17:59:57.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T17:59:57.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T17:59:59.807 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:00:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T18:00:00.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T18:00:00.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:00:00.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:00:00.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:00:00.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:00:00.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:00:00.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.3 2024-06-11T18:00:00.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:00:00.350 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T18:00:00.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T18:00:00.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:00:00.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:00:00.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:00:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:00.387+0000 7f0a0da057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:00.393 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:00.387+0000 7f0a0da057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:00.394 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:00.391+0000 7f0a0da057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:00.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:00.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:00:00.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:00.927+0000 7f0a0da057c0 -1 Falling back to public interface 2024-06-11T18:00:01.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:02.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:00:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:00:02.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:03.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:03.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:03.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:00:03.397 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:03.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:03.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:00:03.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:03.948 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:03.943+0000 7f0a0da057c0 -1 osd.3 69 log_to_monitors true 2024-06-11T18:00:04.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:04.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:04.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:00:04.729 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:04.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:04.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/2289477564,v1:127.0.0.1:6827/2289477564] [v2:127.0.0.1:6828/2289477564,v1:127.0.0.1:6829/2289477564] exists,up 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:00:05.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:00:05.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:00:05.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:00:05.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:00:05.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:05.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:00:06.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836568 2024-06-11T18:00:06.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836568 2024-06-11T18:00:06.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568' 2024-06-11T18:00:06.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.095 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:00:06.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673047 2024-06-11T18:00:06.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673047 2024-06-11T18:00:06.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047' 2024-06-11T18:00:06.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:00:06.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509518 2024-06-11T18:00:06.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509518 2024-06-11T18:00:06.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518' 2024-06-11T18:00:06.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:00:06.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612610 2024-06-11T18:00:06.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612610 2024-06-11T18:00:06.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610' 2024-06-11T18:00:06.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:00:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182456 2024-06-11T18:00:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182456 2024-06-11T18:00:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456' 2024-06-11T18:00:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:00:06.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018928 2024-06-11T18:00:06.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018928 2024-06-11T18:00:06.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456 5-128849018928' 2024-06-11T18:00:06.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:06.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:00:07.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855398 2024-06-11T18:00:07.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855398 2024-06-11T18:00:07.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456 5-128849018928 6-150323855398' 2024-06-11T18:00:07.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:07.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:00:07.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691874 2024-06-11T18:00:07.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691874 2024-06-11T18:00:07.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456 5-128849018928 6-150323855398 7-171798691874' 2024-06-11T18:00:07.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:07.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:00:07.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528341 2024-06-11T18:00:07.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528341 2024-06-11T18:00:07.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456 5-128849018928 6-150323855398 7-171798691874 8-193273528341' 2024-06-11T18:00:07.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:07.401 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:00:07.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364815 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364815 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673047 2-64424509518 3-313532612610 4-107374182456 5-128849018928 6-150323855398 7-171798691874 8-193273528341 9-214748364815' 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836568 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:07.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:00:07.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836568 2024-06-11T18:00:07.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:07.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836568 2024-06-11T18:00:07.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-06-11T18:00:07.550 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836568 2024-06-11T18:00:07.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:00:07.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836569 -lt 21474836568 2024-06-11T18:00:07.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:07.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673047 2024-06-11T18:00:07.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:07.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:00:07.865 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673047 2024-06-11T18:00:07.865 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:07.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673047 2024-06-11T18:00:07.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-06-11T18:00:07.867 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673047 2024-06-11T18:00:07.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:00:08.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673047 -lt 42949673047 2024-06-11T18:00:08.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:08.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:08.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509518 2024-06-11T18:00:08.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:00:08.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509518 2024-06-11T18:00:08.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:08.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509518 2024-06-11T18:00:08.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-06-11T18:00:08.258 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509518 2024-06-11T18:00:08.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:00:08.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509518 -lt 64424509518 2024-06-11T18:00:08.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:08.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612610 2024-06-11T18:00:08.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:08.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:00:08.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:08.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612610 2024-06-11T18:00:08.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612610 2024-06-11T18:00:08.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-06-11T18:00:08.607 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 313532612610 2024-06-11T18:00:08.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:00:08.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612610 -lt 313532612610 2024-06-11T18:00:08.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:08.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182456 2024-06-11T18:00:08.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:08.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:00:08.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182456 2024-06-11T18:00:08.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:08.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182456 2024-06-11T18:00:08.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-06-11T18:00:08.947 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182456 2024-06-11T18:00:08.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:00:09.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182456 -lt 107374182456 2024-06-11T18:00:09.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:09.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018928 2024-06-11T18:00:09.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:09.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:00:09.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018928 2024-06-11T18:00:09.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018928 2024-06-11T18:00:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-06-11T18:00:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 128849018928 2024-06-11T18:00:09.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:00:09.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018928 -lt 128849018928 2024-06-11T18:00:09.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:09.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855398 2024-06-11T18:00:09.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:09.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:00:09.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855398 2024-06-11T18:00:09.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:09.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855398 2024-06-11T18:00:09.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-06-11T18:00:09.617 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855398 2024-06-11T18:00:09.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:00:09.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855399 -lt 150323855398 2024-06-11T18:00:09.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:09.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691874 2024-06-11T18:00:09.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:09.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:00:09.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691874 2024-06-11T18:00:09.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:09.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691874 2024-06-11T18:00:09.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-06-11T18:00:09.974 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691874 2024-06-11T18:00:09.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:00:10.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691874 -lt 171798691874 2024-06-11T18:00:10.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:10.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:10.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528341 2024-06-11T18:00:10.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:00:10.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528341 2024-06-11T18:00:10.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:10.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528341 2024-06-11T18:00:10.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-06-11T18:00:10.309 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 193273528341 2024-06-11T18:00:10.309 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:00:10.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528341 -lt 193273528341 2024-06-11T18:00:10.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:10.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-214748364815 2024-06-11T18:00:10.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:10.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:00:10.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:10.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-214748364815 2024-06-11T18:00:10.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364815 2024-06-11T18:00:10.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 214748364815' 2024-06-11T18:00:10.666 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 214748364815 2024-06-11T18:00:10.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:00:10.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364815 -lt 214748364815 2024-06-11T18:00:10.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:00:10.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:10.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:11.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:00:11.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:00:11.432 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:00:11.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:00:11.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:00:11.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:00:11.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:00:11.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:00:11.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:00:11.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:00:11.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:11.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:12.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:00:12.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:00:12.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:00:12.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T18:00:12.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:00:12.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:00:12.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T18:00:12.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:00:12.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T18:00:12.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T18:00:12.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:00:12.492 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:00:12.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:12.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:12.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:12.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:12.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:12.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:00:13.151 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:00:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:13.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:13.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:13.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:13.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:13.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:13.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:59:45.335648+0000 '>' 2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:13.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:00:14.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:00:14.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:14.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:14.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:14.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:14.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:14.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:14.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T17:59:45.335648+0000 '>' 2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:14.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:00:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:00:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:15.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:15.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:15.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:15.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:15.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:13.944805+0000 '>' 2024-06-11T17:59:45.335648+0000 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:00:16.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:00:16.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:00:16.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:00:16.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:00:16.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:00:16.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:16.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:00:16.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:00:16.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T18:00:16.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:00:16.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:00:16.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T18:00:16.950 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T18:00:16.950 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T18:00:16.950 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:00:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:00:17.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:00:17.245 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:00:17.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:00:17.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T18:00:17.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T18:00:17.247 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.9 2024-06-11T18:00:17.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:00:17.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T18:00:17.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T18:00:17.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:00:17.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:00:17.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:00:17.283 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:17.278+0000 7f1fe38e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:17.326+0000 7f1fe38e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:17.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:17.326+0000 7f1fe38e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:17.876 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:17.870+0000 7f1fe38e77c0 -1 Falling back to public interface 2024-06-11T18:00:17.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:17.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:18.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:19.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:19.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:19.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:00:19.398 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:19.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:19.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:19.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:20.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:20.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:20.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:00:20.738 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:20.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:20.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:20.924 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:20.918+0000 7f1fe38e77c0 -1 osd.9 74 log_to_monitors true 2024-06-11T18:00:21.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:22.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:22.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:22.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:00:22.075 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:22.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:22.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/3941034448,v1:127.0.0.1:6875/3941034448] [v2:127.0.0.1:6876/3941034448,v1:127.0.0.1:6877/3941034448] exists,up b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:00:22.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:00:22.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:00:22.793 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:00:22.793 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:00:22.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:00:22.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:00:22.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:00:22.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:00:22.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:00:22.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:00:22.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:00:22.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:00:22.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T18:00:23.377 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T18:00:23.378 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T18:00:23.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:00:23.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:23.378 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:00:23.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836573 2024-06-11T18:00:23.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836573 2024-06-11T18:00:23.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573' 2024-06-11T18:00:23.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:23.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:00:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673051 2024-06-11T18:00:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673051 2024-06-11T18:00:24.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051' 2024-06-11T18:00:24.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:24.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:00:24.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509522 2024-06-11T18:00:24.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509522 2024-06-11T18:00:24.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522' 2024-06-11T18:00:24.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:24.294 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:00:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612615 2024-06-11T18:00:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612615 2024-06-11T18:00:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615' 2024-06-11T18:00:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:24.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:00:24.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182460 2024-06-11T18:00:24.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182460 2024-06-11T18:00:24.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460' 2024-06-11T18:00:24.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:24.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:00:25.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018933 2024-06-11T18:00:25.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018933 2024-06-11T18:00:25.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460 5-128849018933' 2024-06-11T18:00:25.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:25.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:00:25.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855403 2024-06-11T18:00:25.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855403 2024-06-11T18:00:25.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460 5-128849018933 6-150323855403' 2024-06-11T18:00:25.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:25.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:00:25.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691879 2024-06-11T18:00:25.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691879 2024-06-11T18:00:25.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460 5-128849018933 6-150323855403 7-171798691879' 2024-06-11T18:00:25.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:25.401 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:00:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528345 2024-06-11T18:00:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528345 2024-06-11T18:00:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460 5-128849018933 6-150323855403 7-171798691879 8-193273528345' 2024-06-11T18:00:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:25.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:00:25.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449090 2024-06-11T18:00:25.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449090 2024-06-11T18:00:25.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509522 3-313532612615 4-107374182460 5-128849018933 6-150323855403 7-171798691879 8-193273528345 9-335007449090' 2024-06-11T18:00:25.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:25.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836573 2024-06-11T18:00:25.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:00:25.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836573 2024-06-11T18:00:25.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836573 2024-06-11T18:00:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-06-11T18:00:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836573 2024-06-11T18:00:25.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:00:26.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836573 -lt 21474836573 2024-06-11T18:00:26.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:26.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673051 2024-06-11T18:00:26.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:26.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:00:26.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673051 2024-06-11T18:00:26.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673051 2024-06-11T18:00:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-06-11T18:00:26.027 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673051 2024-06-11T18:00:26.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:00:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673051 -lt 42949673051 2024-06-11T18:00:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:26.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509522 2024-06-11T18:00:26.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:00:26.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509522 2024-06-11T18:00:26.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:26.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509522 2024-06-11T18:00:26.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-06-11T18:00:26.356 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509522 2024-06-11T18:00:26.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:00:26.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509522 -lt 64424509522 2024-06-11T18:00:26.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:26.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612615 2024-06-11T18:00:26.714 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:26.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:00:26.716 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612615 2024-06-11T18:00:26.716 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:26.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612615 2024-06-11T18:00:26.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-06-11T18:00:26.717 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 313532612615 2024-06-11T18:00:26.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:00:27.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612615 -lt 313532612615 2024-06-11T18:00:27.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:27.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182460 2024-06-11T18:00:27.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:27.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:00:27.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182460 2024-06-11T18:00:27.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:27.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182460 2024-06-11T18:00:27.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-06-11T18:00:27.056 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182460 2024-06-11T18:00:27.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:00:27.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182461 -lt 107374182460 2024-06-11T18:00:27.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:27.440 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018933 2024-06-11T18:00:27.440 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:27.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:00:27.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018933 2024-06-11T18:00:27.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:27.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018933 2024-06-11T18:00:27.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018933' 2024-06-11T18:00:27.443 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 128849018933 2024-06-11T18:00:27.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:00:27.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018933 -lt 128849018933 2024-06-11T18:00:27.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:27.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855403 2024-06-11T18:00:27.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:27.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:00:27.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855403 2024-06-11T18:00:27.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:27.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855403 2024-06-11T18:00:27.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-06-11T18:00:27.805 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855403 2024-06-11T18:00:27.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:00:28.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855403 -lt 150323855403 2024-06-11T18:00:28.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:28.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691879 2024-06-11T18:00:28.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:28.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:00:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691879 2024-06-11T18:00:28.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691879 2024-06-11T18:00:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-06-11T18:00:28.237 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691879 2024-06-11T18:00:28.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:00:28.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691879 -lt 171798691879 2024-06-11T18:00:28.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:28.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528345 2024-06-11T18:00:28.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:28.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:00:28.563 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528345 2024-06-11T18:00:28.563 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:28.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528345 2024-06-11T18:00:28.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-06-11T18:00:28.564 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 193273528345 2024-06-11T18:00:28.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:00:28.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528346 -lt 193273528345 2024-06-11T18:00:28.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:28.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-335007449090 2024-06-11T18:00:28.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:28.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:00:28.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-335007449090 2024-06-11T18:00:28.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:28.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449090 2024-06-11T18:00:28.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-06-11T18:00:28.937 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 335007449090 2024-06-11T18:00:28.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:00:29.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449091 -lt 335007449090 2024-06-11T18:00:29.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:00:29.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:29.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:29.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:00:29.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:00:29.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:00:29.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:00:29.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:00:29.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:00:29.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:00:29.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:00:30.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:00:30.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:00:30.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:30.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:30.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:00:30.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:00:30.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:00:30.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:00:30.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:00:30.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:00:30.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:00:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:00:30.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T18:00:33.846 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T18:00:34.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T18:00:34.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T18:00:34.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:00:34.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:00:34.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:00:34.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:00:34.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:00:34.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:00:34.384 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:00:34.384 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:00:34.384 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:00:34.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:00:34.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T18:00:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T18:00:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.5 2024-06-11T18:00:34.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:00:34.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T18:00:34.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T18:00:34.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:00:34.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:00:34.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:00:34.421 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:34.418+0000 7f99e48187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:34.424 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:34.422+0000 7f99e48187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:34.425 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:34.426+0000 7f99e48187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:34.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:00:34.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:34.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:00:34.967 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:34.966+0000 7f99e48187c0 -1 Falling back to public interface 2024-06-11T18:00:35.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:36.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:36.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:36.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:00:36.099 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:36.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:36.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:00:36.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:00:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:37.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:00:37.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:38.146 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:38.142+0000 7f99e48187c0 -1 osd.5 80 log_to_monitors true 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:00:39.647 INFO:tasks.workunit.client.0.smithi012.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/3452759639,v1:127.0.0.1:6843/3452759639] [v2:127.0.0.1:6844/3452759639,v1:127.0.0.1:6845/3452759639] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:00:39.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:00:39.649 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:00:39.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:00:39.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:00:39.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:00:39.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:00:40.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T18:00:40.276 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T18:00:40.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:00:40.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:40.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:00:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836577 2024-06-11T18:00:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836577 2024-06-11T18:00:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577' 2024-06-11T18:00:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:40.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:00:40.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673055 2024-06-11T18:00:40.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673055 2024-06-11T18:00:40.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055' 2024-06-11T18:00:40.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:40.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:00:41.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509527 2024-06-11T18:00:41.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509527 2024-06-11T18:00:41.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527' 2024-06-11T18:00:41.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:41.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:00:41.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612619 2024-06-11T18:00:41.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612619 2024-06-11T18:00:41.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619' 2024-06-11T18:00:41.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:41.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:00:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182465 2024-06-11T18:00:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182465 2024-06-11T18:00:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465' 2024-06-11T18:00:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:41.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:00:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252866 2024-06-11T18:00:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252866 2024-06-11T18:00:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465 5-360777252866' 2024-06-11T18:00:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:41.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:00:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855407 2024-06-11T18:00:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855407 2024-06-11T18:00:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465 5-360777252866 6-150323855407' 2024-06-11T18:00:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:41.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:00:42.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691883 2024-06-11T18:00:42.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691883 2024-06-11T18:00:42.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465 5-360777252866 6-150323855407 7-171798691883' 2024-06-11T18:00:42.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:42.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:00:42.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528350 2024-06-11T18:00:42.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528350 2024-06-11T18:00:42.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465 5-360777252866 6-150323855407 7-171798691883 8-193273528350' 2024-06-11T18:00:42.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:42.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:00:42.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449095 2024-06-11T18:00:42.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449095 2024-06-11T18:00:42.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509527 3-313532612619 4-107374182465 5-360777252866 6-150323855407 7-171798691883 8-193273528350 9-335007449095' 2024-06-11T18:00:42.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:42.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836577 2024-06-11T18:00:42.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:42.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:00:42.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836577 2024-06-11T18:00:42.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:42.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836577 2024-06-11T18:00:42.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-06-11T18:00:42.358 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836577 2024-06-11T18:00:42.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:00:42.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836577 -lt 21474836577 2024-06-11T18:00:42.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:42.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:42.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673055 2024-06-11T18:00:42.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:00:42.674 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673055 2024-06-11T18:00:42.675 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:42.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673055 2024-06-11T18:00:42.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-06-11T18:00:42.678 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673055 2024-06-11T18:00:42.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:00:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673055 -lt 42949673055 2024-06-11T18:00:43.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:43.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509527 2024-06-11T18:00:43.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:43.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:00:43.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509527 2024-06-11T18:00:43.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:43.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509527 2024-06-11T18:00:43.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-06-11T18:00:43.056 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509527 2024-06-11T18:00:43.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:00:43.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509527 -lt 64424509527 2024-06-11T18:00:43.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:43.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:43.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612619 2024-06-11T18:00:43.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:00:43.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612619 2024-06-11T18:00:43.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:43.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612619 2024-06-11T18:00:43.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-06-11T18:00:43.375 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 313532612619 2024-06-11T18:00:43.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:00:43.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612619 -lt 313532612619 2024-06-11T18:00:43.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:43.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182465 2024-06-11T18:00:43.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:43.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:00:43.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182465 2024-06-11T18:00:43.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:43.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182465 2024-06-11T18:00:43.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-06-11T18:00:43.748 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182465 2024-06-11T18:00:43.749 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:00:44.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182465 -lt 107374182465 2024-06-11T18:00:44.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:44.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-360777252866 2024-06-11T18:00:44.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:44.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:00:44.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-360777252866 2024-06-11T18:00:44.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:44.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252866 2024-06-11T18:00:44.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-06-11T18:00:44.116 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 360777252866 2024-06-11T18:00:44.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:00:44.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252866 -lt 360777252866 2024-06-11T18:00:44.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:44.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855407 2024-06-11T18:00:44.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:44.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:00:44.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855407 2024-06-11T18:00:44.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:44.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855407 2024-06-11T18:00:44.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-06-11T18:00:44.426 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855407 2024-06-11T18:00:44.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:00:44.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855407 -lt 150323855407 2024-06-11T18:00:44.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:44.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:44.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691883 2024-06-11T18:00:44.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:00:44.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691883 2024-06-11T18:00:44.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:44.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691883 2024-06-11T18:00:44.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-06-11T18:00:44.794 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691883 2024-06-11T18:00:44.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:00:45.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691883 -lt 171798691883 2024-06-11T18:00:45.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:45.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528350 2024-06-11T18:00:45.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:45.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:00:45.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528350 2024-06-11T18:00:45.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:45.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528350 2024-06-11T18:00:45.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-06-11T18:00:45.170 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 193273528350 2024-06-11T18:00:45.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:00:45.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528350 -lt 193273528350 2024-06-11T18:00:45.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:00:45.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:00:45.528 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-335007449095 2024-06-11T18:00:45.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:00:45.529 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-335007449095 2024-06-11T18:00:45.529 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:00:45.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449095 2024-06-11T18:00:45.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 335007449095' 2024-06-11T18:00:45.531 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 335007449095 2024-06-11T18:00:45.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:00:45.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449095 -lt 335007449095 2024-06-11T18:00:45.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:00:45.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:45.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:46.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:00:46.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:00:46.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:00:46.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:00:46.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:00:46.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:00:46.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:00:46.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:00:46.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:00:46.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:00:46.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:00:46.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:00:46.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:00:46.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T18:00:46.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:00:47.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-11T18:00:47.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:47.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:47.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:47.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:00:48.007 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T18:00:48.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:48.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:48.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:48.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:48.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:48.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:48.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:13.944805+0000 '>' 2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:48.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:00:49.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:00:49.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:49.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:49.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:49.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:49.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:49.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:49.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:13.944805+0000 '>' 2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:49.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:00:50.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:00:50.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:00:50.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:00:50.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:00:50.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:00:50.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:00:50.686 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:00:51.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:48.828649+0000 '>' 2024-06-11T18:00:13.944805+0000 2024-06-11T18:00:51.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:00:51.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:51.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:00:51.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:00:51.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:00:51.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:00:51.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:00:52.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:00:52.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T18:00:52.864 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T18:00:52.864 INFO:tasks.workunit.client.0.smithi012.stdout:hinfo_key 2024-06-11T18:00:52.865 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T18:00:53.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T18:00:53.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:00:53.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:00:53.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T18:00:53.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:00:53.178 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:00:53.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:00:53.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:00:53.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:00:53.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:00:53.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:00:53.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:00:53.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:00:53.181 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:00:53.182 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:00:53.182 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:00:53.182 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:00:53.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.9 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T18:00:53.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:00:53.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:00:53.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:00:53.217 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:53.214+0000 7f8927f097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:53.262 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:53.262+0000 7f8927f097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:53.264 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:53.262+0000 7f8927f097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:00:53.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:53.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:53.820 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:53.818+0000 7f8927f097c0 -1 Falling back to public interface 2024-06-11T18:00:54.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:55.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:55.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:55.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:00:55.087 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:55.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:55.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:55.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:56.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:56.687 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:00:56.686+0000 7f8927f097c0 -1 osd.9 86 log_to_monitors true 2024-06-11T18:00:56.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:00:57.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:00:57.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:00:57.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:00:57.813 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:57.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:00:57.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6874/1855358066,v1:127.0.0.1:6875/1855358066] [v2:127.0.0.1:6876/1855358066,v1:127.0.0.1:6877/1855358066] exists,up b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:00:58.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:00:58.186 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:00:58.186 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:00:58.186 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:00:58.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:00:58.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:00:58.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:00:58.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:4 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:5 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:6 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:7 2024-06-11T18:00:58.788 INFO:tasks.workunit.client.0.smithi012.stderr:8 2024-06-11T18:00:58.789 INFO:tasks.workunit.client.0.smithi012.stderr:9' 2024-06-11T18:00:58.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:00:58.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:58.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:00:59.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836582 2024-06-11T18:00:59.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836582 2024-06-11T18:00:59.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582' 2024-06-11T18:00:59.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:59.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:00:59.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673060 2024-06-11T18:00:59.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673060 2024-06-11T18:00:59.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060' 2024-06-11T18:00:59.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:59.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:00:59.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509531 2024-06-11T18:00:59.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509531 2024-06-11T18:00:59.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531' 2024-06-11T18:00:59.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:00:59.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:01:00.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612624 2024-06-11T18:01:00.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612624 2024-06-11T18:01:00.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624' 2024-06-11T18:01:00.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:01:00.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182470 2024-06-11T18:01:00.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182470 2024-06-11T18:01:00.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470' 2024-06-11T18:01:00.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:01:00.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252871 2024-06-11T18:01:00.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252871 2024-06-11T18:01:00.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470 5-360777252871' 2024-06-11T18:01:00.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:01:00.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855412 2024-06-11T18:01:00.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855412 2024-06-11T18:01:00.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470 5-360777252871 6-150323855412' 2024-06-11T18:01:00.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:01:00.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691888 2024-06-11T18:01:00.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691888 2024-06-11T18:01:00.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470 5-360777252871 6-150323855412 7-171798691888' 2024-06-11T18:01:00.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:01:00.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528354 2024-06-11T18:01:00.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528354 2024-06-11T18:01:00.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470 5-360777252871 6-150323855412 7-171798691888 8-193273528354' 2024-06-11T18:01:00.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:00.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:01:01.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056642 2024-06-11T18:01:01.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056642 2024-06-11T18:01:01.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509531 3-313532612624 4-107374182470 5-360777252871 6-150323855412 7-171798691888 8-193273528354 9-386547056642' 2024-06-11T18:01:01.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:01.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836582 2024-06-11T18:01:01.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:01.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:01:01.183 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836582 2024-06-11T18:01:01.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:01.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836582 2024-06-11T18:01:01.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-06-11T18:01:01.186 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836582 2024-06-11T18:01:01.186 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:01.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836582 -lt 21474836582 2024-06-11T18:01:01.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:01.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673060 2024-06-11T18:01:01.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:01.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:01:01.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673060 2024-06-11T18:01:01.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:01.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673060 2024-06-11T18:01:01.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-06-11T18:01:01.521 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949673060 2024-06-11T18:01:01.521 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:01:01.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673060 -lt 42949673060 2024-06-11T18:01:01.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:01.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509531 2024-06-11T18:01:01.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:01.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:01:01.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509531 2024-06-11T18:01:01.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:01.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509531 2024-06-11T18:01:01.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-06-11T18:01:01.869 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509531 2024-06-11T18:01:01.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:01:02.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509532 -lt 64424509531 2024-06-11T18:01:02.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:02.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612624 2024-06-11T18:01:02.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:02.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:01:02.188 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612624 2024-06-11T18:01:02.188 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:02.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612624 2024-06-11T18:01:02.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-06-11T18:01:02.190 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.3 seq 313532612624 2024-06-11T18:01:02.190 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:01:02.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612624 -lt 313532612624 2024-06-11T18:01:02.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:02.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182470 2024-06-11T18:01:02.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:02.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:01:02.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182470 2024-06-11T18:01:02.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:02.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182470 2024-06-11T18:01:02.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-06-11T18:01:02.581 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.4 seq 107374182470 2024-06-11T18:01:02.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:01:02.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182470 -lt 107374182470 2024-06-11T18:01:02.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:02.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-360777252871 2024-06-11T18:01:02.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:01:02.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-360777252871 2024-06-11T18:01:02.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:02.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252871 2024-06-11T18:01:02.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-06-11T18:01:02.947 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.5 seq 360777252871 2024-06-11T18:01:02.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:01:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252871 -lt 360777252871 2024-06-11T18:01:03.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:03.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855412 2024-06-11T18:01:03.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:03.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:01:03.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855412 2024-06-11T18:01:03.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:03.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855412 2024-06-11T18:01:03.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-06-11T18:01:03.279 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.6 seq 150323855412 2024-06-11T18:01:03.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:01:03.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855412 -lt 150323855412 2024-06-11T18:01:03.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:03.629 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691888 2024-06-11T18:01:03.629 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:03.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:01:03.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691888 2024-06-11T18:01:03.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:03.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691888 2024-06-11T18:01:03.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-06-11T18:01:03.633 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.7 seq 171798691888 2024-06-11T18:01:03.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:01:03.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691888 -lt 171798691888 2024-06-11T18:01:03.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:03.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:03.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528354 2024-06-11T18:01:03.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:01:03.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528354 2024-06-11T18:01:03.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:03.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528354 2024-06-11T18:01:03.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-06-11T18:01:03.991 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.8 seq 193273528354 2024-06-11T18:01:03.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:01:04.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528355 -lt 193273528354 2024-06-11T18:01:04.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:04.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-386547056642 2024-06-11T18:01:04.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:04.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:01:04.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-386547056642 2024-06-11T18:01:04.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:04.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056642 2024-06-11T18:01:04.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-06-11T18:01:04.345 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.9 seq 386547056642 2024-06-11T18:01:04.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:01:04.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 386547056643 -lt 386547056642 2024-06-11T18:01:04.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:01:04.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:04.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:05.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:01:05.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:01:05.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:01:05.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:01:05.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:01:05.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:05.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:05.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:01:05.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:01:05.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:01:05.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:01:05.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:01:05.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-11T18:01:05.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T18:01:05.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 258685"' 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 258685' 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:05.884 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 258687"' 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 258687' 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 258685 258687' 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 258685 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/258688: /' 2024-06-11T18:01:05.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T18:01:05.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/258690: /' 2024-06-11T18:01:10.431 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T18:01:10.432 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-11T18:01:10.433 INFO:tasks.workunit.client.0.smithi012.stderr:258690: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:10.434 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:10.436 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:10.436 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:10.437 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: start osd.9 2024-06-11T18:01:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:01:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T18:01:10.491 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:10.492 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:10.493 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T18:01:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T18:01:10.496 INFO:tasks.workunit.client.0.smithi012.stderr:258688: start osd.5 2024-06-11T18:01:10.496 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2024-06-11T18:01:10.470+0000 7f3a361417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2024-06-11T18:01:10.474+0000 7f3a361417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2024-06-11T18:01:10.474+0000 7f3a361417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.338 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 0 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2024-06-11T18:01:11.010+0000 7f3a361417c0 -1 Falling back to public interface 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.339 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 1 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.340 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2024-06-11T18:01:14.446+0000 7f3a361417c0 -1 osd.9 91 log_to_monitors true 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 3 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258690: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 92 last_c258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T18:01:15.341 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2024-06-11T18:01:10.522+0000 7f8f551607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2024-06-11T18:01:10.526+0000 7f8f551607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2024-06-11T18:01:10.526+0000 7f8f551607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:15.342 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 0 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2024-06-11T18:01:11.078+0000 7f8f551607c0 -1 Falling back to public interface 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 1 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.343 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2024-06-11T18:01:14.458+0000 7f8f551607c0 -1 osd.5 92 log_to_monitors true 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 3 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:15.344 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [90,91) [v2:127.0.0.1:6842/952294273,v1:127.0.0.1:6843/952294273] [v2:127.0.0.1:6844/952294273,v1:127.0.0.1:6845/952294273] exists,up b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:17.137 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:17.138 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 1 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 2 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 3 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 4 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 5 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 6 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 7 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 8 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: 9' 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:17.139 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836587 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836587 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587' 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673065 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673065 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065' 2024-06-11T18:01:17.140 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/stas.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 4 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: osd.5 up in weight 1 up_from 96 up_thru 84 down_at 92 last_clean_interval [84,92) [v2:127.0.0.1:6874/3291793558,v1:127.0.0.1:6875/3291793558] [v2:127.0.0.1:6876/3291793558,v1:127.0.0.1:6877/3291793558] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 1 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 2 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 3 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 4 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 5 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 6 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 7 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 8 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: 9' 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836588 2024-06-11T18:01:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836588 2024-06-11T18:01:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588' 2024-06-11T18:01:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.970 INFO:tasks.workunit.client.0.smithi012.stderr:258688: ndalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:01:18.970 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509536 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509536 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536' 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612628 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612628 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628' 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182474 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182474 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474' 2024-06-11T18:01:18.971 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860418 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860418 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474 5-412316860418' 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855417 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855417 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474 5-412316860418 6-150323855417' 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691892 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691892 2024-06-11T18:01:18.972 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474 5-412316860418 6-150323855417 7-171798691892' 2024-06-11T18:01:18.973 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:18.973 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:01:19.456 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.cli//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:19.456 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673066 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673066 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066' 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509537 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509537 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537' 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612629 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612629 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629' 2024-06-11T18:01:19.457 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182475 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182475 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475' 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860419 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860419 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475 5-412316860419' 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855418 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855418 2024-06-11T18:01:19.458 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475 5-412316860419 6-150323855418' 2024-06-11T18:01:19.459 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.459 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flushent.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528359 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528359 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474 5-412316860418 6-150323855417 7-171798691892 8-193273528359' 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=408021893122 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 408021893122 2024-06-11T18:01:19.838 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673065 2-64424509536 3-313532612628 4-107374182474 5-412316860418 6-150323855417 7-171798691892 8-193273528359 9-408021893122' 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836587 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836587 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836587 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.0 seq 21474836587 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836588 -lt 21474836587 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:19.839 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673065 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673065 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673065 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.1 seq 42949673065 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673066 -lt 42949673065 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509536 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:19.840 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:01:20.118 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.clien_pg_stats: seq=171798691893 2024-06-11T18:01:20.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691893 2024-06-11T18:01:20.118 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475 5-412316860419 6-150323855418 7-171798691893' 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528360 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528360 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475 5-412316860419 6-150323855418 7-171798691893 8-193273528360' 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=408021893123 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 408021893123 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836588 1-42949673066 2-64424509537 3-313532612629 4-107374182475 5-412316860419 6-150323855418 7-171798691893 8-193273528360 9-408021893123' 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836588 2024-06-11T18:01:20.119 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836588 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836588 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.0 seq 21474836588 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836588 -lt 21474836588 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673066 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673066 2024-06-11T18:01:20.120 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673066 2024-06-11T18:01:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-06-11T18:01:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.1 seq 42949673066 2024-06-11T18:01:20.896 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_t.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509536 2024-06-11T18:01:20.896 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.896 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509536 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.2 seq 64424509536 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509537 -lt 64424509536 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612628 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612628 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.897 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612628 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.3 seq 313532612628 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612629 -lt 313532612628 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182474 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182474 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182474 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.4 seq 107374182474 2024-06-11T18:01:20.898 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182475 -lt 107374182474 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-412316860418 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-412316860418 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:20.899 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860418 2024-06-11T18:01:21.530 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting ostats: ceph osd last-stat-seq 1 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673066 -lt 42949673066 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509537 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509537 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509537 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.2 seq 64424509537 2024-06-11T18:01:21.531 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509537 -lt 64424509537 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612629 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612629 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612629 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612629' 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.3 seq 313532612629 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612630 -lt 313532612629 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:21.532 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182475 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182475 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182475 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.4 seq 107374182475 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182476 -lt 107374182475 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:21.533 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-412316860419 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/csd.5 seq 412316860418' 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.5 seq 412316860418 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860420 -lt 412316860418 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855417 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855417 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855417 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.6 seq 150323855417 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855418 -lt 150323855417 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691892 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:01:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691892 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691892 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.7 seq 171798691892 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691894 -lt 171798691892 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528359 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528359 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528359 2024-06-11T18:01:22.285 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-06-11T18:01:22.286 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.8 seq 193273528359 2024-06-11T18:01:22.286 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:01:22.286 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528360 -lt 193273528359 2024-06-11T18:01:22.540 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_staephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.540 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:01:22.540 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-412316860419 2024-06-11T18:01:22.540 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.540 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860419 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.5 seq 412316860419 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860420 -lt 412316860419 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855418 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855418 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855418 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-06-11T18:01:22.541 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.6 seq 150323855418 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855418 -lt 150323855418 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691893 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691893 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691893 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.7 seq 171798691893 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:01:22.542 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691894 -lt 171798691893 2024-06-11T18:01:22.543 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:22.543 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528360 2024-06-11T18:01:22.543 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:22.543 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:01:22.543 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528360 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: ts: for s in $seqs 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-408021893122 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-408021893122 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=408021893122 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: waiting osd.9 seq 408021893122 2024-06-11T18:01:23.725 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 408021893124 -lt 408021893122 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:01:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:258690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:01:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:01:24.514 INFO:tasks.workunit.client.0.smithi012.stderr:cut -d - -f 2 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528360 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.8 seq 193273528360 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528360 -lt 193273528360 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-408021893123 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-408021893123 2024-06-11T18:01:24.515 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=408021893123 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: waiting osd.9 seq 408021893123 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 408021893124 -lt 408021893123 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:01:24.516 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:258688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 258687 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T18:01:24.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T18:01:24.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T18:01:24.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T18:01:24.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T18:01:24.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T18:01:24.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:01:24.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:01:24.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T18:01:24.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:01:24.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T18:01:24.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T18:01:24.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:01:24.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:01:24.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:24.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:25.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:25.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:01:25.596 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T18:01:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:01:25.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:01:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:25.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:48.828649+0000 '>' 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:25.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:01:26.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:01:26.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:01:26.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:01:26.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:26.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:26.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:26.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:27.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:48.828649+0000 '>' 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:27.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:01:28.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:01:28.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:01:28.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:01:28.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:28.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:28.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:28.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:28.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:48.828649+0000 '>' 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:28.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:01:29.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:01:29.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:01:29.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:01:29.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:29.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:29.626 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:29.626 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:29.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:00:48.828649+0000 '>' 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:29.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:01:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:01:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:01:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:01:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:01:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:01:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:01:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:00:48.828649+0000 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 262117"' 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 262117' 2024-06-11T18:01:31.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 262119"' 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 262119' 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 262117 262119' 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 262117 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/262120: /' 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:01:31.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/262122: /' 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:33.501 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:33.502 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: _ 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: hinfo_key 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: snapset 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:33.503 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:33.505 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:33.506 INFO:tasks.workunit.client.0.smithi012.stderr:262120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T18:01:33.507 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262120: start osd.5 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:33.508 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:33.509 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: _ 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: hinfo_key 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: snapset 2024-06-11T18:01:33.510 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:33.511 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:33.512 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:33.512 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:33.513 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:33.514 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-11T18:01:33.515 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-11T18:01:33.515 INFO:tasks.workunit.client.0.smithi012.stderr:262122: start osd.9 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2024-06-11T18:01:33.554+0000 7f5df73187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2024-06-11T18:01:33.558+0000 7f5df73187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.572 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2024-06-11T18:01:33.562+0000 7f5df73187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 0 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2024-06-11T18:01:34.114+0000 7f5df73187c0 -1 Falling back to public interface 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.573 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 1 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2 2024-06-11T18:01:38.574 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 3 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2024-06-11T18:01:38.254+0000 7f5df73187c0 -1 osd.5 98 log_to_monitors true 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-11T18:01:38.575 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2024-06-11T18:01:33.550+0000 7f1876f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2024-06-11T18:01:33.558+0000 7f1876f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2024-06-11T18:01:33.558+0000 7f1876f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:38.576 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 0 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2024-06-11T18:01:34.126+0000 7f1876f3d7c0 -1 Falling back to public interface 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 1 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 3 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:38.578 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2024-06-11T18:01:38.218+0000 7f1876f3d7c0 -1 osd.9 98 log_to_monitors true 2024-06-11T18:01:41.075 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 4 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: osd.9 up in weight 1 up_from 102 up_thru 0 down_at 99 last_clean_interval [95,98) [v2:127.0.0.1:6874/3047281917,v1:127.0.0.1:6875/3047281917] [v2:127.0.0.1:6876/3047281917,v1:127.0.0.1:6877/3047281917] exists,up b1d7fcb0-3d3f-41ba-b2c2-d8f48938a508 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:41.077 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 1 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 2 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 3 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 4 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 5 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 6 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 7 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 8 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: 9' 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836594 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836594 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594' 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ideph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 4 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: osd.5 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6842/3988759543,v1:127.0.0.1:6843/3988759543] [v2:127.0.0.1:6844/3988759543,v1:127.0.0.1:6845/3988759543] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:41.080 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 1 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 2 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 3 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 4 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 5 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 6 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 7 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 8 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: 9' 2024-06-11T18:01:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836593 2024-06-11T18:01:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836593 2024-06-11T18:01:41.082 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593' 2024-06-11T18:01:42.336 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.336 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673072 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673072 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072' 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509544 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509544 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544' 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612635 2024-06-11T18:01:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612635 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635' 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182481 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182481 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481' 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664195 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664195 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481 5-438086664195' 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855424 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855424 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481 5-438086664195 6-150323855424' 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shs 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673073 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673073 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073' 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:01:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509543 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509543 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543' 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=313532612636 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 313532612636 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636' 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182482 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182482 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482' 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.340 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664194 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664194 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482 5-438086664194' 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855423 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855423 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482 5-438086664194 6-150323855423' 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:01:44.296 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2:2218: flush_pg_stats: seq=171798691900 2024-06-11T18:01:44.296 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691900 2024-06-11T18:01:44.296 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481 5-438086664195 6-150323855424 7-171798691900' 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528366 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528366 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481 5-438086664195 6-150323855424 7-171798691900 8-193273528366' 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664194 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664194 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673072 2-64424509544 3-313532612635 4-107374182481 5-438086664195 6-150323855424 7-171798691900 8-193273528366 9-438086664194' 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836593 2024-06-11T18:01:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836593 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836593 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.0 seq 21474836593 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836592 -lt 21474836593 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836595 -lt 21474836593 2024-06-11T18:01:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673072 2024-06-11T18:01:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:01:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673072 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:26218: flush_pg_stats: seq=171798691899 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691899 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482 5-438086664194 6-150323855423 7-171798691899' 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528367 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528367 2024-06-11T18:01:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482 5-438086664194 6-150323855423 7-171798691899 8-193273528367' 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664195 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664195 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509543 3-313532612636 4-107374182482 5-438086664194 6-150323855423 7-171798691899 8-193273528367 9-438086664195' 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836594 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836594 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:44.302 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836594 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.0 seq 21474836594 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836592 -lt 21474836594 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836595 -lt 21474836594 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673073 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:01:44.303 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673073 2024-06-11T18:01:45.345 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.345 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673073 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.1 seq 42949673073 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673073 -lt 42949673073 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509543 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509543 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509543 2024-06-11T18:01:45.346 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.2 seq 64424509543 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509544 -lt 64424509543 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612636 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612636 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612636 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612636' 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.3 seq 313532612636 2024-06-11T18:01:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612637 -lt 313532612636 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182482 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182482 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182482 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.4 seq 107374182482 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cep2120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673072 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673072' 2024-06-11T18:01:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.1 seq 42949673072 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673073 -lt 42949673072 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509544 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509544 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509544 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.2 seq 64424509544 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509544 -lt 64424509544 2024-06-11T18:01:45.349 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-313532612635 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-313532612635 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=313532612635 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.3 seq 313532612635 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 313532612637 -lt 313532612635 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182481 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182481 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182481 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-06-11T18:01:45.351 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.4 seq 107374182481 2024-06-11T18:01:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/chtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:01:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182482 -lt 107374182482 2024-06-11T18:01:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-438086664194 2024-06-11T18:01:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:01:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-438086664194 2024-06-11T18:01:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664194 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 438086664194' 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.5 seq 438086664194 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664196 -lt 438086664194 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855423 2024-06-11T18:01:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:01:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855423 2024-06-11T18:01:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855423 2024-06-11T18:01:46.693 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-06-11T18:01:46.693 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.6 seq 150323855423 2024-06-11T18:01:46.693 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:01:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855425 -lt 150323855423 2024-06-11T18:01:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691899 2024-06-11T18:01:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.694 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:01:46.695 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691899 2024-06-11T18:01:46.695 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.695 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691899 2024-06-11T18:01:46.695 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-06-11T18:01:46.696 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.7 seq 171798691899 2024-06-11T18:01:46.696 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:01:46.696 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691900 -lt 171798691899 2024-06-11T18:01:46.696 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.696 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:01:46.697 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182482 -lt 107374182481 2024-06-11T18:01:46.697 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.697 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-438086664195 2024-06-11T18:01:46.697 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:01:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-438086664195 2024-06-11T18:01:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664195 2024-06-11T18:01:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 438086664195' 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.5 seq 438086664195 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664196 -lt 438086664195 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855424 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855424 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855424 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-06-11T18:01:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.6 seq 150323855424 2024-06-11T18:01:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:01:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855425 -lt 150323855424 2024-06-11T18:01:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691900 2024-06-11T18:01:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:01:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691900 2024-06-11T18:01:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691900 2024-06-11T18:01:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-06-11T18:01:46.703 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.7 seq 171798691900 2024-06-11T18:01:46.703 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:01:46.703 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691900 -lt 171798691900 2024-06-11T18:01:46.703 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528366 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528366 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528366 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.8 seq 193273528366 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528367 -lt 193273528366 2024-06-11T18:01:48.145 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664194 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664194 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664194 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: waiting osd.9 seq 438086664194 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664196 -lt 438086664194 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:01:48.147 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:01:48.151 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceplpers.sh:2227: flush_pg_stats: echo 8-193273528367 2024-06-11T18:01:48.151 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528367 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528367 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.8 seq 193273528367 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528367 -lt 193273528367 2024-06-11T18:01:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664195 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664195 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664195 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664195' 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: waiting osd.9 seq 438086664195 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664196 -lt 438086664195 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:01:48.153 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:01:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:01:48.530 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph h --format json status 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:262120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:262120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:01:48.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 262119 2024-06-11T18:01:48.535 INFO:tasks.workunit.client.0.smithi012.stderr:--format json status 2024-06-11T18:01:48.535 INFO:tasks.workunit.client.0.smithi012.stderr:262122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:01:48.535 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:01:48.536 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:01:48.536 INFO:tasks.workunit.client.0.smithi012.stderr:262122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:01:48.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T18:01:48.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:01:48.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:01:48.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-11T18:01:48.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-11T18:01:48.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-11T18:01:48.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-11T18:01:48.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 265082"' 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 265082' 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 265083"' 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 265083' 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-11T18:01:48.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 265082 265083' 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 265082 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/265085: /' 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:48.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/265087: /' 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T18:01:52.729 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T18:01:52.730 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:52.731 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:52.734 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:52.735 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: start osd.3 2024-06-11T18:01:52.736 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:01:52.939 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T18:01:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:01:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:01:52.943 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:01:52.944 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: start osd.5 2024-06-11T18:01:52.945 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2024-06-11T18:01:52.766+0000 7f6becc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2024-06-11T18:01:52.770+0000 7f6becc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2024-06-11T18:01:52.770+0000 7f6becc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 0 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:01:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2024-06-11T18:01:53.326+0000 7f6becc867c0 -1 Falling back to public interface 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 1 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:01:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2024-06-11T18:01:56.642+0000 7f6becc867c0 -1 osd.3 104 log_to_monitors true 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 3 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.466 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:01:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:265085: osd.3 up in weight 1 up_from 108 up_thru 102 down_at 105 l265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T18:01:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T18:01:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2024-06-11T18:01:52.974+0000 7f75ee6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2024-06-11T18:01:52.974+0000 7f75ee6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2024-06-11T18:01:52.978+0000 7f75ee6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 0 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2024-06-11T18:01:53.526+0000 7f75ee6d87c0 -1 Falling back to public interface 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 1 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2024-06-11T18:01:56.790+0000 7f75ee6d87c0 -1 osd.5 104 log_to_monitors true 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 3 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:01:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265087: osd.5 up in weight 1 up_from 108 up_thru 102 down_at 105 last_clean_interval [73,104) [v2:127.0.0.1:6826/697018237,v1:127.0.0.1:6827/697018237] [v2:127.0.0.1:6828/697018237,v1:127.0.0.1:6829/697018237] exists,up 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:58.818 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:58.819 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 1 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 2 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 3 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 4 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 5 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 6 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 7 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 8 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: 9' 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:58.820 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836599 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836599 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599' 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673077 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673077 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077' 2024-06-11T18:01:58.821 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:59.441 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/ast_clean_interval [102,104) [v2:127.0.0.1:6842/808697465,v1:127.0.0.1:6843/808697465] [v2:127.0.0.1:6844/808697465,v1:127.0.0.1:6845/808697465] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T18:01:59.441 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:01:59.442 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 1 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 2 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 3 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 4 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 5 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 6 2024-06-11T18:01:59.443 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 7 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 8 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: 9' 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836600 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836600 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600' 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673078 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673078 2024-06-11T18:01:59.444 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078' 2024-06-11T18:01:59.445 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:02:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509548 2024-06-11T18:02:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509548 2024-06-11T18:02:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548' 2024-06-11T18:02:00.451 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=463856467970 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 463856467970 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970' 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182486 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182486 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486' 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=463856467970 2024-06-11T18:02:00.452 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 463856467970 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486 5-463856467970' 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855429 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855429 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486 5-463856467970 6-150323855429' 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691904 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691904 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486 5-463856467970 6-150323855429 7-171798691904' 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.453 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:02:00.753 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clo/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509549 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509549 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549' 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=463856467971 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 463856467971 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971' 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:02:00.754 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182488 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182488 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488' 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=463856467971 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 463856467971 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488 5-463856467971' 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855430 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855430 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488 5-463856467971 6-150323855430' 2024-06-11T18:02:00.755 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691906 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691906 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488 5-463856467971 6-150323855430 7-171798691906' 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:00.756 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:02:01.330 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clne.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528371 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528371 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486 5-463856467970 6-150323855429 7-171798691904 8-193273528371' 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664200 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664200 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836599 1-42949673077 2-64424509548 3-463856467970 4-107374182486 5-463856467970 6-150323855429 7-171798691904 8-193273528371 9-438086664200' 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836599 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:02:01.331 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836599 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836599 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.0 seq 21474836599 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836600 -lt 21474836599 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673077 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673077 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:01.332 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673077 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.1 seq 42949673077 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673078 -lt 42949673077 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509548 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.333 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:02:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/cloneone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528372 2024-06-11T18:02:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528372 2024-06-11T18:02:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488 5-463856467971 6-150323855430 7-171798691906 8-193273528372' 2024-06-11T18:02:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:01.612 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664201 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664201 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836600 1-42949673078 2-64424509549 3-463856467971 4-107374182488 5-463856467971 6-150323855430 7-171798691906 8-193273528372 9-438086664201' 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836600 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836600 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836600 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2024-06-11T18:02:01.613 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.0 seq 21474836600 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836600 -lt 21474836600 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673078 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673078 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673078 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673078' 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.1 seq 42949673078 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:02:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673078 -lt 42949673078 2024-06-11T18:02:01.615 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:01.615 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509549 2024-06-11T18:02:01.615 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:01.615 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:02:02.380 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clon.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509548 2024-06-11T18:02:02.380 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509548 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.2 seq 64424509548 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509550 -lt 64424509548 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-463856467970 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-463856467970 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=463856467970 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 463856467970' 2024-06-11T18:02:02.381 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.3 seq 463856467970 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 463856467971 -lt 463856467970 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182486 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182486 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182486 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.4 seq 107374182486 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:02:02.382 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182488 -lt 107374182486 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-463856467970 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-463856467970 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.383 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=463856467970 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waie.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509549 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509549 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.2 seq 64424509549 2024-06-11T18:02:02.732 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509550 -lt 64424509549 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-463856467971 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-463856467971 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=463856467971 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 463856467971' 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.3 seq 463856467971 2024-06-11T18:02:02.733 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 463856467971 -lt 463856467971 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182488 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182488 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182488 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-06-11T18:02:02.734 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.4 seq 107374182488 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182488 -lt 107374182488 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-463856467971 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-463856467971 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:02.735 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=463856467971 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'wating osd.5 seq 463856467970' 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.5 seq 463856467970 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 463856467971 -lt 463856467970 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855429 2024-06-11T18:02:03.833 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855429 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855429 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.6 seq 150323855429 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855430 -lt 150323855429 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691904 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:03.834 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691904 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691904 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.7 seq 171798691904 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691906 -lt 171798691904 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528371 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528371 2024-06-11T18:02:03.835 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528371 2024-06-11T18:02:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-06-11T18:02:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.8 seq 193273528371 2024-06-11T18:02:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:02:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528373 -lt 193273528371 2024-06-11T18:02:04.191 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_iting osd.5 seq 463856467971' 2024-06-11T18:02:04.191 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.5 seq 463856467971 2024-06-11T18:02:04.191 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:02:04.191 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 463856467971 -lt 463856467971 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855430 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855430 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855430 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.6 seq 150323855430 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855430 -lt 150323855430 2024-06-11T18:02:04.192 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691906 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691906 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691906 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.7 seq 171798691906 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691906 -lt 171798691906 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528372 2024-06-11T18:02:04.193 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528372 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528372 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.8 seq 193273528372 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:02:04.194 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528373 -lt 193273528372 2024-06-11T18:02:05.386 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flushpg_stats: for s in $seqs 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664200 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664200 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664200 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664200' 2024-06-11T18:02:05.387 INFO:tasks.workunit.client.0.smithi012.stderr:265085: waiting osd.9 seq 438086664200 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664201 -lt 438086664200 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:02:05.388 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:265085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:02:05.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:02:05.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 265083 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:_pg_stats: for s in $seqs 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664201 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664201 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664201 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664201' 2024-06-11T18:02:05.717 INFO:tasks.workunit.client.0.smithi012.stderr:265087: waiting osd.9 seq 438086664201 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664201 -lt 438086664201 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:02:05.718 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:265087: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-11T18:02:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T18:02:05.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-11T18:02:05.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:02:05.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:02:05.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-11T18:02:05.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:02:06.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-11T18:02:06.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-11T18:02:06.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:02:06.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:02:06.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:06.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:06.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:06.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:06.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:06.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:02:06.723 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:02:06.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:06.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:06.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:06.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:06.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:06.742 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:07.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:07.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:02:08.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:02:08.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:08.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:08.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:08.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:08.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:08.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:08.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:08.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:02:09.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:02:09.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:09.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:09.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:09.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:09.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:09.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:09.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:09.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:02:10.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:02:10.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:10.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:10.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:10.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:10.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:10.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:11.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:11.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:02:12.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:02:12.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:12.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:12.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:12.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:12.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:12.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:12.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:01:25.859139+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:12.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:02:13.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:02:13.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:02:13.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:02:13.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:02:13.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:02:13.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:02:13.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:02:13.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:02:07.603558+0000 '>' 2024-06-11T18:01:25.859139+0000 2024-06-11T18:02:13.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:02:13.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-11T18:02:13.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T18:02:13.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:02:13.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:02:13.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 268513"' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 268513' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 268515"' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/268516: /' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 268515' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 268513 268515' 2024-06-11T18:02:13.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-11T18:02:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:02:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 268513 2024-06-11T18:02:13.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/268518: /' 2024-06-11T18:02:14.863 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.863 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:02:14.863 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-11T18:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:02:14.864 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:02:14.865 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:02:14.866 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: _ 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: hinfo_key 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: snapset 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:02:14.867 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-11T18:02:14.868 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:02:14.868 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:02:14.868 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:02:14.868 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:14.869 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:14.870 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:14.870 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:02:14.870 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-11T18:02:14.871 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-11T18:02:14.872 INFO:tasks.workunit.client.0.smithi012.stderr:268518: start osd.5 2024-06-11T18:02:14.879 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.879 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:02:14.879 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-11T18:02:14.879 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:02:14.879 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:02:14.880 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: _ 2024-06-11T18:02:14.881 INFO:tasks.workunit.client.0.smithi012.stderr:268516: hinfo_key 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: snapset 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-11T18:02:14.882 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-11T18:02:14.883 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:02:14.884 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-11T18:02:14.885 INFO:tasks.workunit.client.0.smithi012.stderr:268516: start osd.3 2024-06-11T18:02:20.049 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-overy_ops 2024-06-11T18:02:20.049 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-11T18:02:20.049 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2024-06-11T18:02:14.902+0000 7f802aad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2024-06-11T18:02:14.906+0000 7f802aad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2024-06-11T18:02:14.910+0000 7f802aad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2024-06-11T18:02:15.446+0000 7f802aad17c0 -1 Falling back to public interface 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-11T18:02:20.050 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 0 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 1 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.051 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2024-06-11T18:02:19.265+0000 7f802aad17c0 -1 osd.5 109 log_to_monitors true 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 3 2024-06-11T18:02:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:02:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2024-06-11T18:02:14.930+0000 7f67fa1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2024-06-11T18:02:15.034+0000 7f67fa1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2024-06-11T18:02:15.038+0000 7f67fa1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 0 2024-06-11T18:02:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2024-06-11T18:02:15.602+0000 7f67fa1437c0 -1 Falling back to public interface 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 1 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.055 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 3 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:20.056 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.clientceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 4 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: osd.5 up in weight 1 up_from 113 up_thru 108 down_at 110 last_clean_interval [108,109) [v2:127.0.0.1:6826/134237730,v1:127.0.0.1:6827/134237730] [v2:127.0.0.1:6828/134237730,v1:127.0.0.1:6829/134237730] exists,up b1f8f08b-bf3f-4f62-b386-9c73197152a2 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:02:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:02:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 1 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 2 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 3 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 4 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 5 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 6 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 7 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 8 2024-06-11T18:02:22.257 INFO:tasks.workunit.client.0.smithi012.stderr:268518: 9' 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836606 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836606 2024-06-11T18:02:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606' 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2024-06-11T18:02:20.093+0000 7f67fa1437c0 -1 osd.3 109 log_to_monitors true 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:02:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 4 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 5 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: osd.3 up in weight 1 up_from 116 up_thru 108 down_at 110 last_clean_interval [108,109) [v2:127.0.0.1:6842/1741126329,v1:127.0.0.1:6843/1741126329] [v2:127.0.0.1:6844/1741126329,v1:127.0.0.1:6845/1741126329] exists,up 2608169b-5ddc-4b3b-ac20-c2c2b4da8804 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:02:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:02:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 1 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 2 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 3 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 4 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 5 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 6 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 7 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 8 2024-06-11T18:02:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:268516: 9' 2024-06-11T18:02:24.694 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standaloids 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673084 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673084 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084' 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509555 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509555 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555' 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.695 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=498216206338 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 498216206338 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338' 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182493 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182493 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493' 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=485331304450 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 485331304450 2024-06-11T18:02:24.696 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493 5-485331304450' 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855436 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855436 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493 5-485331304450 6-150323855436' 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:24.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:02:25.545 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691911 2024-06-11T18:02:25.545 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691911 2024-06-11T18:02:25.545 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493 5-485331304450 6-150323855436 7-171798691911' 2024-06-11T18:02:25.545 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528378 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528378 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493 5-485331304450 6-150323855436 7-171798691911 8-193273528378' 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.546 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664207 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664207 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673084 2-64424509555 3-498216206338 4-107374182493 5-485331304450 6-150323855436 7-171798691911 8-193273528378 9-438086664207' 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836606 2024-06-11T18:02:25.547 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836606 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836606 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836606' 2024-06-11T18:02:25.548 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.0 seq 21474836606 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836607 -lt 21474836606 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673084 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:25.549 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:02:25.550 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673084 2024-06-11T18:02:25.550 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:25.550 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673084 2024-06-11T18:02:25.550 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673084' 2024-06-11T18:02:25.550 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.1 seq 42949673084 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223ne/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836607 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836607 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607' 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673085 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673085 2024-06-11T18:02:25.697 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085' 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509556 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509556 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556' 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=498216206339 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 498216206339 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339' 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-11T18:02:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182495 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182495 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495' 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=485331304452 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 485331304452 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495 5-485331304452' 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:25.699 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-11T18:02:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855437 2024-06-11T18:02:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855437 2024-06-11T18:02:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495 5-485331304452 6-150323855437' 2024-06-11T18:02:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-11T18:02:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691913 2024-06-11T18:02:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691913 2024-06-11T18:02:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495 5-485331304452 6-150323855437 7-171798691913' 2024-06-11T18:02:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-11T18:02:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528379 2024-06-11T18:02:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528379 2024-06-11T18:02:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495 5-485331304452 6-150323855437 7-171798691913 8-193273528379' 2024-06-11T18:02:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=438086664208 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 438086664208 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836607 1-42949673085 2-64424509556 3-498216206339 4-107374182495 5-485331304452 6-150323855437 7-171798691913 8-193273528379 9-438086664208' 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836607 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:02:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836607 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836607 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.0 seq 21474836607 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836607 -lt 21474836607 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673085 2024-06-11T18:02:27.004 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/0: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673085 -lt 42949673084 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509555 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509555 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.005 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509555 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509555' 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.2 seq 64424509555 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509556 -lt 64424509555 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-498216206338 2024-06-11T18:02:27.006 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-498216206338 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=498216206338 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.3 seq 498216206338 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 498216206338 -lt 498216206338 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182493 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182493 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182493 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.4 seq 107374182493 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182493 -lt 107374182493 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.008 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.589 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.589 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:02:27.589 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673085 2024-06-11T18:02:27.589 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673085 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673085' 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.1 seq 42949673085 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673085 -lt 42949673085 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509556 2024-06-11T18:02:27.590 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509556 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509556 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.2 seq 64424509556 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509556 -lt 64424509556 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-498216206339 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-498216206339 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:27.591 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=498216206339 2024-06-11T18:02:27.592 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-06-11T18:02:27.592 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.3 seq 498216206339 2024-06-11T18:02:27.592 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-11T18:02:27.593 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 498216206339 -lt 498216206339 2024-06-11T18:02:27.593 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:27.594 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182495 2024-06-11T18:02:27.594 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:27.594 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-06-11T18:02:27.594 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182495 2024-06-11T18:02:27.594 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.064 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-485331304450 2024-06-11T18:02:28.064 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-485331304450 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=485331304450 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 485331304450' 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.5 seq 485331304450 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 485331304452 -lt 485331304450 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855436 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855436 2024-06-11T18:02:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855436 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855436' 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.6 seq 150323855436 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855437 -lt 150323855436 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691911 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691911 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691911 2024-06-11T18:02:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691911' 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.7 seq 171798691911 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691913 -lt 171798691911 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528378 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:02:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528378 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flushclient.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182495 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182495' 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.4 seq 107374182495 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182495 -lt 107374182495 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-485331304452 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.886 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-485331304452 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=485331304452 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 485331304452' 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.5 seq 485331304452 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 485331304452 -lt 485331304452 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 6-150323855437 2024-06-11T18:02:28.887 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=6 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 6-150323855437 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855437 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.6 seq 150323855437' 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.6 seq 150323855437 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855437 -lt 150323855437 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 7-171798691913 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=7 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 7-171798691913 2024-06-11T18:02:28.888 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:28.889 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691913 2024-06-11T18:02:28.889 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.7 seq 171798691913' 2024-06-11T18:02:28.889 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.7 seq 171798691913 2024-06-11T18:02:28.889 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clon_pg_stats: cut -d - -f 2 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528378 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.8 seq 193273528378 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528380 -lt 193273528378 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664207 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:29.839 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664207 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664207 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664207' 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: waiting osd.9 seq 438086664207 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664208 -lt 438086664207 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:02:29.840 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:268518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:02:29.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:02:30.334 INFO:tasks.workunit.client.0.smithi012.stderr:e.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691913 -lt 171798691913 2024-06-11T18:02:30.334 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:30.334 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 8-193273528379 2024-06-11T18:02:30.334 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:30.334 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=8 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 8-193273528379 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528379 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.8 seq 193273528379 2024-06-11T18:02:30.335 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528380 -lt 193273528379 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 9-438086664208 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=9 2024-06-11T18:02:30.336 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 9-438086664208 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=438086664208 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.9 seq 438086664208' 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: waiting osd.9 seq 438086664208 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 438086664209 -lt 438086664208 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:30.337 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:02:30.338 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:02:30.719 INFO:tasks.workunit.client.0.smithi012.stderr:268516: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 268515 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-11T18:02:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:02:30.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:02:30.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:02:30.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:02:30.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:02:30.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:02:30.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:02:30.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:02:30.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:02:30.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:02:30.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:02:31.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:02:31.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:02:31.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:02:31.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:02:31.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:02:31.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:02:31.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:02:31.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:02:31.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:02:31.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:02:31.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:02:31.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:02:31.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:02:31.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:02:31.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:02:31.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:02:31.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:02:31.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:02:31.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:02:31.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:02:31.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:02:31.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:02:31.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:02:31.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:02:31.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:02:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:02:31.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:02:31.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:02:31.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:02:31.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:02:31.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:02:31.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:02:31.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:02:31.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:02:31.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:02:31.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:02:31.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:02:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:02:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:02:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:02:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:02:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-06-11T18:02:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:02:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-11T18:02:31.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:02:31.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:02:31.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:02:31.668 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:02:31.668 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.668 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:31.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-11T18:02:31.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:02:31.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:02:31.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:02:31.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:02:31.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.871 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:02:31.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:02:31.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:02:31.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:02:31.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:02:31.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:02:31.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:02:31.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:02:31.990 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:02:31.990 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:02:31.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:02:31.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:02:32.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-06-11T18:02:32.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:02:32.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:02:32.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:02:32.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:02:32.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:02:32.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:02:32.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:02:32.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:02:32.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:02:32.977 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:02:32.977 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:32.977 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:32.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:32.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:02:32.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:02:33.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-06-11T18:02:33.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:02:33.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:02:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:02:33.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:02:33.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:02:33.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47cd0f2c-509c-4524-9219-86b8122e2477 2024-06-11T18:02:33.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 47cd0f2c-509c-4524-9219-86b8122e2477' 2024-06-11T18:02:33.022 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 47cd0f2c-509c-4524-9219-86b8122e2477 2024-06-11T18:02:33.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:02:33.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5kWhmLbxSAhAAJ2vWMOOk5vUiH9wGaukpYQ== 2024-06-11T18:02:33.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5kWhmLbxSAhAAJ2vWMOOk5vUiH9wGaukpYQ=="}' 2024-06-11T18:02:33.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47cd0f2c-509c-4524-9219-86b8122e2477 -i td/osd-scrub-repair/0/new.json 2024-06-11T18:02:33.286 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:02:33.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:02:33.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5kWhmLbxSAhAAJ2vWMOOk5vUiH9wGaukpYQ== --osd-uuid 47cd0f2c-509c-4524-9219-86b8122e2477 2024-06-11T18:02:33.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.333+0000 7f571b0e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:33.337 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.333+0000 7f571b0e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:33.338 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.337+0000 7f571b0e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:33.338 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.337+0000 7f571b0e77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:02:33.339 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.337+0000 7f571b0e77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:02:33.339 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.337+0000 7f571b0e77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:02:33.429 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:33.425+0000 7f571b0e77c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:02:39.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:02:39.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:02:39.205 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:02:39.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:02:39.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:02:39.615 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:02:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:02:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:02:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:02:39.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:02:39.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:02:39.649 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:39.645+0000 7ff2b48fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:39.675 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:39.673+0000 7ff2b48fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:39.677 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:39.673+0000 7ff2b48fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:02:39.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:02:39.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:02:39.954 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:02:39.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:40.224 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:40.221+0000 7ff2b48fb7c0 -1 Falling back to public interface 2024-06-11T18:02:40.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:41.323 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:02:41.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:02:41.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:41.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:41.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:42.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:42.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:42.649 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:02:42.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:02:42.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:42.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:42.768 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:02:42.765+0000 7ff2b48fb7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:02:43.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:44.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:44.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:44.044 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:02:44.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:02:44.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:44.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:44.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:45.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:45.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:45.488 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:02:45.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:02:45.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:45.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:45.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:46.838 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:02:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:02:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:47.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:48.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:48.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:48.172 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:02:48.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:02:48.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:48.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:48.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:49.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:49.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:49.537 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:02:49.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:02:49.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:49.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:49.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:50.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:50.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:50.875 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:02:50.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:02:50.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:50.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:51.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:52.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:52.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:52.215 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:02:52.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:02:52.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:52.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:52.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:53.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:53.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:53.653 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:02:53.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:02:53.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:53.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:53.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:54.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:54.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:54.988 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:02:54.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:02:54.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:54.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:55.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:56.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:56.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:56.575 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:02:56.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:02:56.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:56.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:57.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:58.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:58.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:58.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:02:58.008 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:02:58.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:58.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:02:58.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:02:59.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:02:59.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:02:59.695 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:02:59.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:02:59.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:02:59.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:00.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:01.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:01.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:01.130 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:03:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:03:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:01.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:02.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:02.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:02.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:03:02.873 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:03:02.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:02.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:03.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:04.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:04.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:04.380 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:03:04.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:03:04.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:04.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:04.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:05.719 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:03:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:03:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:05.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:06.052 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2800962577,v1:127.0.0.1:6803/2800962577] [v2:127.0.0.1:6804/2800962577,v1:127.0.0.1:6805/2800962577] exists,up 47cd0f2c-509c-4524-9219-86b8122e2477 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:03:06.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:03:06.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:03:06.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:03:06.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:03:06.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:03:06.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:03:06.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2ce27246-9967-4cd3-8f5f-308477f88fb5' 2024-06-11T18:03:06.059 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:03:06.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:03:06.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBakWhmBYvvBBAAIgMmWqosGxPvlh6ysENjUA== 2024-06-11T18:03:06.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBakWhmBYvvBBAAIgMmWqosGxPvlh6ysENjUA=="}' 2024-06-11T18:03:06.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ce27246-9967-4cd3-8f5f-308477f88fb5 -i td/osd-scrub-repair/1/new.json 2024-06-11T18:03:06.445 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:03:06.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:03:06.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBakWhmBYvvBBAAIgMmWqosGxPvlh6ysENjUA== --osd-uuid 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:03:06.494 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.493+0000 7fa6e3d717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:06.529 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.525+0000 7fa6e3d717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:06.531 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.529+0000 7fa6e3d717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:06.531 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.529+0000 7fa6e3d717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:03:06.531 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.529+0000 7fa6e3d717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:03:06.532 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.529+0000 7fa6e3d717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:03:06.604 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:06.601+0000 7fa6e3d717c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:03:12.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:03:12.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:03:12.397 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:03:12.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:03:12.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:03:12.812 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:03:12.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:03:12.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:03:12.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:03:12.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:03:12.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:03:12.845 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:12.841+0000 7fbf119bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:12.875 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:12.873+0000 7fbf119bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:12.876 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:12.873+0000 7fbf119bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:13.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:03:13.160 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:03:13.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:13.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:13.436 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:13.433+0000 7fbf119bc7c0 -1 Falling back to public interface 2024-06-11T18:03:13.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:14.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:14.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:14.507 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:03:14.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:03:14.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:14.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:14.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:15.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:15.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:15.843 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:03:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:03:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:15.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:15.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:15.945+0000 7fbf119bc7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:03:16.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:17.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:17.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:17.214 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:03:17.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:03:17.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:17.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:17.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:18.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:18.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:18.548 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:03:18.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:03:18.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:18.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:18.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:19.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:19.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:19.887 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:03:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:03:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:19.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:20.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:21.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:21.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:21.360 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:03:21.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:03:21.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:21.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:21.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:22.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:22.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:22.804 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:03:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:03:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:22.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:23.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:24.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:24.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:24.132 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:03:24.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:03:24.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:24.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:24.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:25.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:25.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:25.655 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:03:25.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:03:25.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:25.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:26.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:27.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:27.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:27.022 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:03:27.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:03:27.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:27.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:27.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:28.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:28.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:28.360 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:03:28.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:03:28.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:28.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:28.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:29.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:29.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:29.819 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:03:29.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:03:29.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:29.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:30.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:31.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:31.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:31.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:03:31.818 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:03:31.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:31.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:32.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:33.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:33.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:33.580 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:03:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:03:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:34.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:35.126 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:03:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:03:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:35.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:36.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:36.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:36.914 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:03:36.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:03:36.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:36.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:37.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:38.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:38.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:38.468 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:03:38.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:03:38.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:38.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:38.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:39.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:39.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:39.874 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:03:39.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:03:39.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:39.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:03:40.215 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3971836928,v1:127.0.0.1:6811/3971836928] [v2:127.0.0.1:6812/3971836928,v1:127.0.0.1:6813/3971836928] exists,up 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:03:40.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:03:40.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:03:40.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:03:40.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-06-11T18:03:40.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:03:40.549 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:03:40.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:03:40.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:03:40.999 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:03:41.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:03:42.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:03:43.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-06-11T18:03:43.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:03:43.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:03:43.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:03:43.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:03:43.253 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:03:43.253 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:03:43.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:03:43.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:03:43.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:03:43.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:03:43.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:03:43.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:03:43.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:03:43.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:03:43.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:03:43.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:03:43.725 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:03:43.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:03:43.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:03:43.725 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:03:43.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-06-11T18:03:43.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-06-11T18:03:43.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-06-11T18:03:43.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:03:43.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:03:44.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672963 2024-06-11T18:03:44.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672963 2024-06-11T18:03:44.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493 1-42949672963' 2024-06-11T18:03:44.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:03:44.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-06-11T18:03:44.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:03:44.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:03:44.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-06-11T18:03:44.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:03:44.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-06-11T18:03:44.034 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836493 2024-06-11T18:03:44.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-06-11T18:03:44.035 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:03:44.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-06-11T18:03:44.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:03:44.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:03:44.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672963 2024-06-11T18:03:44.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:03:44.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:03:44.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672963 2024-06-11T18:03:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672963 2024-06-11T18:03:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-06-11T18:03:44.368 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672963 2024-06-11T18:03:44.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:03:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672962 -lt 42949672963 2024-06-11T18:03:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:03:45.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:03:45.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:03:46.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672964 -lt 42949672963 2024-06-11T18:03:46.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:03:46.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:03:46.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:03:46.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:03:46.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:03:46.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:03:46.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:03:46.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:03:46.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:03:47.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:03:47.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:03:47.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:03:47.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:03:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:03:47.592 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:03:47.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:03:48.397 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:03:48.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:03:48.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:03:48.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-11T18:03:48.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-06-11T18:03:48.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-11T18:03:48.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-11T18:03:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-11T18:03:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-11T18:03:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T18:03:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-11T18:03:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:03:49.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T18:03:49.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-11T18:03:49.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T18:03:49.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:03:49.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:03:49.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:03:49.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:03:49.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-11T18:03:49.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:03:49.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:03:49.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:03:49.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:03:49.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:03:49.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-11T18:03:51.061 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T18:03:51.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:03:51.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:03:51.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:03:51.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:03:51.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:03:51.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:03:51.599 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:03:51.600 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:03:51.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:03:51.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:03:51.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:03:51.602 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:03:51.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:03:51.602 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:03:51.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:03:51.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:03:51.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:03:51.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:03:51.642 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:51.641+0000 7fd8acd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:51.645 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:51.641+0000 7fd8acd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:51.646 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:51.645+0000 7fd8acd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:03:51.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:51.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:52.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:52.173+0000 7fd8acd5e7c0 -1 Falling back to public interface 2024-06-11T18:03:52.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:53.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:53.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:53.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:03:53.332 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:03:53.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:53.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:53.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:54.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:54.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:54.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:03:54.663 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:03:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:54.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:03:54.713+0000 7fd8acd5e7c0 -1 osd.0 20 log_to_monitors true 2024-06-11T18:03:55.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:03:56.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:03:56.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:03:56.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:03:56.078 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:03:56.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:03:56.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/2149521057,v1:127.0.0.1:6803/2149521057] [v2:127.0.0.1:6804/2149521057,v1:127.0.0.1:6805/2149521057] exists,up 47cd0f2c-509c-4524-9219-86b8122e2477 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:03:56.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:03:56.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:03:56.437 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:03:56.437 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:03:56.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:03:56.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:03:56.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:03:56.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:03:56.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:03:56.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:03:56.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:03:56.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:03:56.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:03:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:03:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:03:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:03:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:03:56.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:03:57.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215106 2024-06-11T18:03:57.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215106 2024-06-11T18:03:57.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106' 2024-06-11T18:03:57.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:03:57.045 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:03:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672967 2024-06-11T18:03:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672967 2024-06-11T18:03:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106 1-42949672967' 2024-06-11T18:03:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:03:57.190 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215106 2024-06-11T18:03:57.190 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:03:57.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:03:57.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215106 2024-06-11T18:03:57.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:03:57.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215106 2024-06-11T18:03:57.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-11T18:03:57.193 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 103079215106 2024-06-11T18:03:57.193 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:03:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 103079215106 2024-06-11T18:03:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:03:58.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:03:58.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:03:58.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-11T18:03:58.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:03:58.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672967 2024-06-11T18:03:58.870 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:03:58.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:03:58.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672967 2024-06-11T18:03:58.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:03:58.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672967 2024-06-11T18:03:58.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-11T18:03:58.875 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672967 2024-06-11T18:03:58.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:03:59.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-11T18:03:59.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:03:59.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:03:59.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:03:59.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:03:59.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:03:59.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:03:59.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:03:59.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:03:59.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:03:59.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:00.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:04:00.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:04:00.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:04:00.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-11T18:04:00.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-11T18:04:00.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:04:00.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-11T18:04:00.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:04:00.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-11T18:04:00.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-11T18:04:00.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-06-11T18:04:00.660 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-06-11T18:04:00.660 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:00.660 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:00.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:00.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:01.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:01.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-06-11T18:04:01.308 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-11T18:04:01.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:01.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.3 2024-06-11T18:04:01.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:01.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:04:01.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:04:01.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:01.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:01.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:01.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:01.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:01.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:01.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:03:40.948066+0000 '>' 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:01.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:02.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:02.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:02.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:02.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:02.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:02.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:02.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:02.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:03:40.948066+0000 '>' 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:02.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:03.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:03.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:03.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:03.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:03.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:03.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:03.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:04.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:03:40.948066+0000 '>' 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:04.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:05.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:05.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:05.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:05.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:05.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:05.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:05.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:05.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:03:40.948066+0000 '>' 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:05.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:06.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:06.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:06.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:06.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:06.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:06.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:06.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:01.747394+0000 '>' 2024-06-11T18:03:40.948066+0000 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:04:07.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:04:07.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:04:07.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:07.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T18:04:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T18:04:07.556 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T18:04:07.556 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T18:04:07.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:04:07.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:04:07.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:04:07.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:07.845 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:04:07.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:04:07.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:04:07.847 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:04:07.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:04:07.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:04:07.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:04:07.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:04:07.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:04:07.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:04:07.882 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:07.881+0000 7f1f707017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:07.885 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:07.881+0000 7f1f707017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:07.886 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:07.885+0000 7f1f707017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:08.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:08.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:08.435 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:08.433+0000 7f1f707017c0 -1 Falling back to public interface 2024-06-11T18:04:08.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:09.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:09.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:09.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:04:09.588 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:04:09.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:09.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:09.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:10.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:10.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:10.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:04:10.929 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:04:10.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:10.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:11.208 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:11.205+0000 7f1f707017c0 -1 osd.1 26 log_to_monitors true 2024-06-11T18:04:11.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:12.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:12.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:12.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:04:12.267 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:04:12.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:12.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1882361913,v1:127.0.0.1:6811/1882361913] [v2:127.0.0.1:6812/1882361913,v1:127.0.0.1:6813/1882361913] exists,up 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:04:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:04:12.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:04:12.610 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:04:12.610 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:04:12.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:04:12.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:04:12.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:04:12.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:04:12.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:04:12.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:04:12.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:04:12.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:04:12.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:04:13.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:04:13.049 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:04:13.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:04:13.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:13.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:04:13.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215110 2024-06-11T18:04:13.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215110 2024-06-11T18:04:13.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215110' 2024-06-11T18:04:13.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:13.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215110 1-128849018882' 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215110 2024-06-11T18:04:13.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:13.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:04:13.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215110 2024-06-11T18:04:13.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:13.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215110 2024-06-11T18:04:13.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-06-11T18:04:13.367 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 103079215110 2024-06-11T18:04:13.367 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:13.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215109 -lt 103079215110 2024-06-11T18:04:13.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:04:14.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:04:14.684 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:15.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215110 -lt 103079215110 2024-06-11T18:04:15.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:15.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-128849018882 2024-06-11T18:04:15.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:15.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:04:15.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-128849018882 2024-06-11T18:04:15.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:15.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T18:04:15.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-06-11T18:04:15.023 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 128849018882 2024-06-11T18:04:15.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:04:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T18:04:15.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:04:15.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:15.370 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:15.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:04:15.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:04:16.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:04:16.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:04:16.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:16.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:16.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:04:16.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:04:16.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:04:16.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:04:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:04:16.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-06-11T18:04:16.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-11T18:04:16.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-11T18:04:16.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-11T18:04:16.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:04:16.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:04:16.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:04:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:04:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-11T18:04:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:04:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:04:17.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:04:17.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:04:17.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:17.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-11T18:04:18.538 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:04:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:04:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:04:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:04:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:04:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:04:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:19.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:19.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:19.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:04:19.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:04:19.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:04:19.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:04:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:04:19.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:04:19.078 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:04:19.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:04:19.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:04:19.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:04:19.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:04:19.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:04:19.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:04:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:19.113+0000 7fbb35d237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:19.118 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:19.117+0000 7fbb35d237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:19.120 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:19.117+0000 7fbb35d237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:19.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:04:19.419 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:04:19.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:19.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:19.651 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:19.649+0000 7fbb35d237c0 -1 Falling back to public interface 2024-06-11T18:04:19.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:20.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:20.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:20.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:04:20.758 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:04:20.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:20.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:21.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:22.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:22.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:22.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:04:22.097 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:04:22.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:22.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:22.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:22.361+0000 7fbb35d237c0 -1 osd.1 31 log_to_monitors true 2024-06-11T18:04:22.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:23.830 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/2963552600,v1:127.0.0.1:6811/2963552600] [v2:127.0.0.1:6812/2963552600,v1:127.0.0.1:6813/2963552600] exists,up 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:04:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:04:23.832 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:04:23.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:04:23.832 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:04:23.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:04:23.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:04:23.833 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:04:23.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:04:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:04:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:04:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:04:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:24.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:04:24.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215113 2024-06-11T18:04:24.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215113 2024-06-11T18:04:24.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215113' 2024-06-11T18:04:24.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:24.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:04:24.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855362 2024-06-11T18:04:24.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855362 2024-06-11T18:04:24.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215113 1-150323855362' 2024-06-11T18:04:24.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:24.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215113 2024-06-11T18:04:24.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:24.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:04:24.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215113 2024-06-11T18:04:24.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:24.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215113 2024-06-11T18:04:24.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215113' 2024-06-11T18:04:24.589 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 103079215113 2024-06-11T18:04:24.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:24.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215112 -lt 103079215113 2024-06-11T18:04:24.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:04:25.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:04:25.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:26.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215114 -lt 103079215113 2024-06-11T18:04:26.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:26.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:26.234 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855362 2024-06-11T18:04:26.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:04:26.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855362 2024-06-11T18:04:26.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:26.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855362 2024-06-11T18:04:26.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-11T18:04:26.238 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 150323855362 2024-06-11T18:04:26.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:04:26.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-11T18:04:26.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:04:26.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:26.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:04:26.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:04:26.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:04:26.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:04:27.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:04:27.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:04:27.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:27.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:27.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:04:27.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:04:27.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:04:27.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-11T18:04:27.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-11T18:04:27.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-11T18:04:27.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-11T18:04:27.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:04:28.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-11T18:04:28.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-11T18:04:28.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-06-11T18:04:28.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-06-11T18:04:28.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:28.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:28.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:28.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:28.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:28.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-06-11T18:04:28.731 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-11T18:04:28.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:28.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.3 2024-06-11T18:04:28.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:28.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:04:28.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:04:28.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:28.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:28.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:28.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:28.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:28.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:29.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:01.747394+0000 '>' 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:29.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:30.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:30.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:30.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:30.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:30.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:30.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:30.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:30.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:01.747394+0000 '>' 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:30.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:31.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:31.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:31.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:31.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:31.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:31.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:31.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:31.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:01.747394+0000 '>' 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:31.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:32.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:32.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:32.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:32.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:32.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:32.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:32.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:01.747394+0000 '>' 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:33.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:04:34.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:04:34.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:04:34.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-11T18:04:34.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-11T18:04:34.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:04:34.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:04:34.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:04:29.288129+0000 '>' 2024-06-11T18:04:01.747394+0000 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:34.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:04:34.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:34.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-11T18:04:34.914 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-11T18:04:34.914 INFO:tasks.workunit.client.0.smithi012.stdout:_ 2024-06-11T18:04:34.914 INFO:tasks.workunit.client.0.smithi012.stdout:snapset 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:04:35.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:35.213 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:35.214 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:04:35.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:04:35.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:04:35.216 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:04:35.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:04:35.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:04:35.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:04:35.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:04:35.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:04:35.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:04:35.250 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:35.248+0000 7f319d4867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:35.253 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:35.248+0000 7f319d4867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:35.255 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:35.252+0000 7f319d4867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:04:35.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:35.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:04:35.551 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:04:35.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:35.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:35.796 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:35.792+0000 7f319d4867c0 -1 Falling back to public interface 2024-06-11T18:04:35.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:36.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:36.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:36.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:04:36.909 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:04:36.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:36.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:37.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:04:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:04:38.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:38.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:38.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:38.737 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:38.732+0000 7f319d4867c0 -1 osd.1 37 log_to_monitors true 2024-06-11T18:04:39.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:39.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:39.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:04:39.566 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:04:39.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:39.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:04:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/1388444598,v1:127.0.0.1:6811/1388444598] [v2:127.0.0.1:6812/1388444598,v1:127.0.0.1:6813/1388444598] exists,up 2ce27246-9967-4cd3-8f5f-308477f88fb5 2024-06-11T18:04:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:04:40.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:04:40.029 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:04:40.029 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:04:40.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:04:40.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:04:40.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:04:40.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:04:40.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:04:40.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:04:40.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:04:40.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:04:40.150 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:04:40.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:04:40.485 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:04:40.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:04:40.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:40.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:04:40.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215118 2024-06-11T18:04:40.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215118 2024-06-11T18:04:40.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215118' 2024-06-11T18:04:40.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:04:40.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:04:40.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=176093659138 2024-06-11T18:04:40.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 176093659138 2024-06-11T18:04:40.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215118 1-176093659138' 2024-06-11T18:04:40.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:40.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:40.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215118 2024-06-11T18:04:40.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:04:40.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:40.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215118 2024-06-11T18:04:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215118 2024-06-11T18:04:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215118' 2024-06-11T18:04:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 103079215118 2024-06-11T18:04:40.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:41.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215116 -lt 103079215118 2024-06-11T18:04:41.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:04:42.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:04:42.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:04:42.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215118 -lt 103079215118 2024-06-11T18:04:42.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:04:42.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-176093659138 2024-06-11T18:04:42.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:04:42.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:04:42.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-176093659138 2024-06-11T18:04:42.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:04:42.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=176093659138 2024-06-11T18:04:42.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-06-11T18:04:42.457 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 176093659138 2024-06-11T18:04:42.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:04:42.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659138 -lt 176093659138 2024-06-11T18:04:42.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:04:42.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:42.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:43.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:04:43.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:04:43.200 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:04:43.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:04:43.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:04:43.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:04:43.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:04:43.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:04:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:04:43.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:04:43.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:04:43.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:04:43.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:04:43.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:04:43.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:04:43.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-11T18:04:43.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-11T18:04:44.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:04:44.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:04:44.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:04:44.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:04:44.001 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:04:44.001 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:04:44.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:04:44.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:04:44.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:04:44.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:04:44.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:04:44.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:04:44.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:04:44.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:04:44.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:04:44.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:04:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:04:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:04:44.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:04:44.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:04:44.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:04:44.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:04:44.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:04:44.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:04:44.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:44.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:44.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:04:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:04:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:04:44.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:04:44.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:04:44.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:04:44.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:04:44.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:04:44.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:04:44.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:04:44.353 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:04:44.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:04:44.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:04:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:04:44.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:04:44.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:04:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:04:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:04:44.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:04:44.360 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:44.360 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:44.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:04:44.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:04:44.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:04:44.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:04:44.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:04:44.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:44.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:44.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:04:44.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:04:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:04:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:04:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:04:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:04:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:04:44.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:04:44.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:44.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:44.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:04:45.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:04:45.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:04:45.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:04:45.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:04:45.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:45.024 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:04:45.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:04:45.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:04:45.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:04:45.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:04:45.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:04:45.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:04:45.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:45.145 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:04:45.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:04:45.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:04:45.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-11T18:04:45.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:04:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:04:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:04:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:04:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:04:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:04:46.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:46.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:46.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:04:46.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:04:46.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:04:46.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:04:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:04:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:04:46.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:04:46.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:04:46.176 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:04:46.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f2ea95ed-6d17-4c91-9547-90a369b22c6e' 2024-06-11T18:04:46.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:04:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+kWhmmqOHCxAALFU1LtU3r2xpAL6kYPvOmQ== 2024-06-11T18:04:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+kWhmmqOHCxAALFU1LtU3r2xpAL6kYPvOmQ=="}' 2024-06-11T18:04:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2ea95ed-6d17-4c91-9547-90a369b22c6e -i td/osd-scrub-repair/0/new.json 2024-06-11T18:04:46.434 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:04:46.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:04:46.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+kWhmmqOHCxAALFU1LtU3r2xpAL6kYPvOmQ== --osd-uuid f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:04:46.483 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.480+0000 7f1ecfd8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:46.485 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.480+0000 7f1ecfd8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:46.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.484+0000 7f1ecfd8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:46.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.484+0000 7f1ecfd8b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:04:46.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.484+0000 7f1ecfd8b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:04:46.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.484+0000 7f1ecfd8b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:04:46.576 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:46.572+0000 7f1ecfd8b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:04:52.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:04:52.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:04:52.037 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:04:52.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:04:52.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:04:52.441 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:04:52.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:04:52.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:04:52.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:04:52.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:04:52.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:04:52.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:52.472+0000 7f95052f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:52.499 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:52.496+0000 7f95052f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:52.500 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:52.496+0000 7f95052f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:04:52.782 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:04:52.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:52.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:53.060 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:53.056+0000 7f95052f47c0 -1 Falling back to public interface 2024-06-11T18:04:53.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:54.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:54.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:54.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:04:54.146 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:04:54.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:54.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:54.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:55.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:55.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:55.482 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:04:55.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:04:55.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:55.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:55.607 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:04:55.604+0000 7f95052f47c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:04:55.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:56.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:56.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:56.817 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:04:56.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:04:56.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:56.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:57.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:58.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:58.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:58.152 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:04:58.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:04:58.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:58.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:04:59.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:04:59.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:04:59.630 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:04:59.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:04:59.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:04:59.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:04:59.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:00.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:00.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:00.967 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:05:00.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:05:00.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:00.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:01.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:02.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:02.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:02.356 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:05:02.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:05:02.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:02.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:02.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:03.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:03.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:03.694 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:05:03.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:05:03.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:03.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:04.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:05.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:05.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:05.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:05:05.029 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:05:05.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:05.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:05.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:06.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:06.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:06.383 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:05:06.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:05:06.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:06.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:06.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:07.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:07.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:07.719 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:05:07.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:05:07.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:07.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:08.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:09.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:09.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:09.515 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:05:09.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:05:09.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:09.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:09.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:10.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:10.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:10.914 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:05:10.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:05:10.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:10.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:11.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:12.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:12.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:12.712 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:05:12.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:05:12.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:12.713 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:13.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:14.049 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:05:14.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:14.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:14.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:05:14.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:14.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:14.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:15.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:15.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:15.788 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:05:15.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:05:15.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:15.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:16.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:17.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:17.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:17.743 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:05:17.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:05:17.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:17.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:05:18.113 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2440556903,v1:127.0.0.1:6803/2440556903] [v2:127.0.0.1:6804/2440556903,v1:127.0.0.1:6805/2440556903] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:05:18.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:05:18.116 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:05:18.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:05:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:05:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:05:18.117 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:05:18.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:05:18.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:05:18.119 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:05:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c0fb9163-8a9f-480a-839a-b8ff3904b239' 2024-06-11T18:05:18.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:05:18.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDekWhmhi42CBAAaBYJY1m+A30liIc/KcTfNQ== 2024-06-11T18:05:18.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDekWhmhi42CBAAaBYJY1m+A30liIc/KcTfNQ=="}' 2024-06-11T18:05:18.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0fb9163-8a9f-480a-839a-b8ff3904b239 -i td/osd-scrub-repair/1/new.json 2024-06-11T18:05:18.509 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:05:18.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:05:18.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDekWhmhi42CBAAaBYJY1m+A30liIc/KcTfNQ== --osd-uuid c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:05:18.566 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.564+0000 7fe5777687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:18.594 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.592+0000 7fe5777687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:18.595 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.592+0000 7fe5777687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:18.596 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.592+0000 7fe5777687c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:05:18.596 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.592+0000 7fe5777687c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:05:18.596 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.592+0000 7fe5777687c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:05:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:18.652+0000 7fe5777687c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:05:24.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:05:24.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:05:24.569 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:05:24.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:05:24.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:05:24.981 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:05:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:05:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:05:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:05:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:05:24.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:05:25.014 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:25.012+0000 7f891a5947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:25.040 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:25.036+0000 7f891a5947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:25.042 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:25.040+0000 7f891a5947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:05:25.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:25.357 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:05:25.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:05:25.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:25.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:25.612 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:25.608+0000 7f891a5947c0 -1 Falling back to public interface 2024-06-11T18:05:25.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:26.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:26.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:26.699 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:05:26.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:05:26.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:26.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:27.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:28.034 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:05:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:05:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:28.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:28.160+0000 7f891a5947c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:05:28.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:29.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:29.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:29.382 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:05:29.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:05:29.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:29.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:29.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:31.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:31.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:31.001 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:05:31.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:05:31.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:31.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:31.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:32.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:32.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:32.402 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:05:32.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:05:32.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:32.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:32.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:33.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:33.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:33.753 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:05:33.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:05:33.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:33.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:35.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:35.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:35.206 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:05:35.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:05:35.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:35.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:35.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:36.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:36.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:36.593 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:05:36.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:05:36.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:36.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:36.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:37.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:37.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:37.994 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:05:37.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:05:37.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:37.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:38.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:39.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:39.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:39.330 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:05:39.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:05:39.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:39.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:39.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:40.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:40.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:40.783 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:05:40.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:05:40.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:40.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:41.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:42.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:42.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:42.429 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:05:42.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:05:42.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:42.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:42.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:43.982 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:05:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:05:43.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:43.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:44.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:45.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:45.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:45.317 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:05:45.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:05:45.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:45.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:46.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:47.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:47.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:47.005 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:05:47.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:05:47.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:47.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:47.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:48.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:48.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:48.456 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:05:48.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:05:48.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:48.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:49.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:50.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:50.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:50.150 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:05:50.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:05:50.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:50.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:50.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:05:51.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:05:51.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:51.964 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:05:51.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:05:51.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:51.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:05:52.293 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/125017139,v1:127.0.0.1:6811/125017139] [v2:127.0.0.1:6812/125017139,v1:127.0.0.1:6813/125017139] exists,up c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:05:52.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:05:52.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:05:52.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:05:52.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:05:52.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:05:52.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:05:52.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:05:52.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:05:52.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bbd7068f-67e0-41de-be9f-c78dd0d005e4' 2024-06-11T18:05:52.300 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:05:52.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:05:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAkmhmVZrIEhAAnkOa94NitISUPVrYSQ6EBQ== 2024-06-11T18:05:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAkmhmVZrIEhAAnkOa94NitISUPVrYSQ6EBQ=="}' 2024-06-11T18:05:52.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbd7068f-67e0-41de-be9f-c78dd0d005e4 -i td/osd-scrub-repair/2/new.json 2024-06-11T18:05:52.705 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:05:52.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T18:05:52.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAkmhmVZrIEhAAnkOa94NitISUPVrYSQ6EBQ== --osd-uuid bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:05:52.750 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.748+0000 7f62487f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:52.781 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.776+0000 7f62487f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:52.783 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.780+0000 7f62487f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:52.784 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.780+0000 7f62487f27c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:05:52.784 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.780+0000 7f62487f27c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:05:52.784 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.780+0000 7f62487f27c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:05:52.857 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:52.852+0000 7f62487f27c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T18:05:58.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T18:05:58.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:05:58.712 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T18:05:58.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T18:05:58.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:05:59.126 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T18:05:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T18:05:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:05:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:05:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:05:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:05:59.160 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:59.156+0000 7f985db3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:59.180 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:59.176+0000 7f985db3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:59.181 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:59.176+0000 7f985db3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:05:59.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:05:59.474 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:05:59.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:05:59.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:05:59.728 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:05:59.724+0000 7f985db3a7c0 -1 Falling back to public interface 2024-06-11T18:05:59.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:00.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:00.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:00.822 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:06:00.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:06:00.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:00.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:01.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:02.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:02.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:02.148 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:06:02.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:06:02.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:02.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:02.288 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:02.284+0000 7f985db3a7c0 -1 osd.2 0 log_to_monitors true 2024-06-11T18:06:02.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:03.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:03.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:03.498 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:06:03.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:06:03.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:03.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:03.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:04.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:04.901 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:06:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:06:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:06.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:06.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:06.306 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:06:06.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:06:06.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:06.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:06.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:07.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:07.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:07.639 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:06:07.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:06:07.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:07.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:07.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:08.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:08.980 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:06:08.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:06:08.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:08.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:09.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:10.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:10.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:10.318 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:06:10.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:06:10.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:10.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:10.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:11.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:11.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:11.655 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:06:11.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:06:11.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:11.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:12.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:13.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:13.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:13.082 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:06:13.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:06:13.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:13.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:13.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:14.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:14.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:14.589 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:06:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:06:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:14.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:15.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:16.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:16.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:16.068 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:06:16.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:06:16.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:16.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:16.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:17.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:17.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:17.753 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:06:17.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:06:17.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:17.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:18.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:19.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:19.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:19.487 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:06:19.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:06:19.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:19.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:20.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:21.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:21.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:21.205 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:06:21.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:06:21.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:21.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:22.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:23.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:23.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:23.010 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:06:23.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:06:23.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:23.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:23.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:24.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:24.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:24.636 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:06:24.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:06:24.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:24.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:24.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:26.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:26.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:26.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:06:26.000 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:06:26.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:26.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:06:26.339 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2893348089,v1:127.0.0.1:6819/2893348089] [v2:127.0.0.1:6820/2893348089,v1:127.0.0.1:6821/2893348089] exists,up bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:06:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:06:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:06:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:06:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-11T18:06:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:06:26.664 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:06:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:06:26.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:06:27.106 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:06:27.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:06:28.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:06:30.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-11T18:06:30.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-11T18:06:30.482 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'foo' created 2024-06-11T18:06:30.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:06:31.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-06-11T18:06:31.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T18:06:31.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T18:06:31.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T18:06:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T18:06:31.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-11T18:06:31.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T18:06:31.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T18:06:32.809 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T18:06:32.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:06:33.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:06:33.835 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:06:33.835 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:06:33.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:06:33.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:06:33.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:06:33.975 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:06:34.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:06:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:06:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:06:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:06:34.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:34.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:06:34.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836501 2024-06-11T18:06:34.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836501 2024-06-11T18:06:34.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501' 2024-06-11T18:06:34.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:34.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:06:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-06-11T18:06:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-06-11T18:06:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672973' 2024-06-11T18:06:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:34.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:06:34.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-06-11T18:06:34.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-06-11T18:06:34.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672973 2-64424509447' 2024-06-11T18:06:34.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:34.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:34.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836501 2024-06-11T18:06:34.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:06:34.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836501 2024-06-11T18:06:34.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:34.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836501 2024-06-11T18:06:34.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-11T18:06:34.822 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836501 2024-06-11T18:06:34.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:06:35.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836501 2024-06-11T18:06:35.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:06:36.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:06:36.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:06:36.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-11T18:06:36.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:36.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-06-11T18:06:36.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:36.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:06:36.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-06-11T18:06:36.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:36.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-06-11T18:06:36.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-11T18:06:36.485 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672973 2024-06-11T18:06:36.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:06:36.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-11T18:06:36.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:36.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-06-11T18:06:36.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:36.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:06:36.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-06-11T18:06:36.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:36.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-06-11T18:06:36.830 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509447 2024-06-11T18:06:36.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-11T18:06:36.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:06:37.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-11T18:06:37.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:06:37.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:37.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:37.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:06:37.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:06:37.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:06:37.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:06:37.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:06:37.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:37.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:06:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:06:38.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:06:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:06:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:06:38.355 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:06:38.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:06:38.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:06:38.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:06:38.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:06:38.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:06:38.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:06:38.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:06:38.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:06:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:06:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:06:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:06:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:06:38.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:38.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:06:38.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-06-11T18:06:38.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-06-11T18:06:38.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-06-11T18:06:38.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:38.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:06:39.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-06-11T18:06:39.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-06-11T18:06:39.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-06-11T18:06:39.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:39.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:06:39.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-06-11T18:06:39.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-06-11T18:06:39.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509449' 2024-06-11T18:06:39.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:39.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-06-11T18:06:39.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:39.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:06:39.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-06-11T18:06:39.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:39.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-06-11T18:06:39.253 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836503 2024-06-11T18:06:39.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-06-11T18:06:39.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:06:39.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-06-11T18:06:39.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:39.574 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-06-11T18:06:39.574 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:39.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:06:39.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:39.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-06-11T18:06:39.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-06-11T18:06:39.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-11T18:06:39.579 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672975 2024-06-11T18:06:39.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:06:39.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-06-11T18:06:39.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:39.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-06-11T18:06:39.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:39.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:06:39.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-06-11T18:06:39.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:39.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-06-11T18:06:39.913 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509449 2024-06-11T18:06:39.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-11T18:06:39.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:06:40.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509449 2024-06-11T18:06:40.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:06:41.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:06:41.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:06:41.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509449 2024-06-11T18:06:41.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:06:41.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:41.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:41.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:06:41.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:06:41.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:06:41.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:06:41.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:06:41.999 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:06:42.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:06:42.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:06:42.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:06:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:06:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:42.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:42.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:06:42.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:06:42.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:06:42.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:06:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:06:43.123 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:06:43.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:06:43.718 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:06:43.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:06:43.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:06:43.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:06:43.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:06:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:06:43.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:06:43.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:06:43.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:06:43.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:06:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:06:45.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:06:45.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:46.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:06:46.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:06:46.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:06:46.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:06:46.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:06:46.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:06:46.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:06:46.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:06:46.964 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:06:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:06:46.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:06:46.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:06:46.966 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:06:46.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:06:46.966 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:06:46.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:06:46.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:06:46.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:06:46.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:06:47.001 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:46.995+0000 7fbf933f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:06:47.004 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:46.999+0000 7fbf933f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:06:47.005 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:46.999+0000 7fbf933f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:06:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:06:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:06:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:06:47.539 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:47.535+0000 7fbf933f27c0 -1 Falling back to public interface 2024-06-11T18:06:47.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:48.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:48.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:48.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:06:48.663 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:06:48.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:48.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:06:48.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:50.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:50.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:50.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:06:50.002 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:06:50.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:06:50.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:50.033 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:06:50.027+0000 7fbf933f27c0 -1 osd.1 33 log_to_monitors true 2024-06-11T18:06:50.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:06:51.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:06:51.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:06:51.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:06:51.447 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:06:51.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:06:51.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:06:51.836 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 34 last_clean_interval [10,33) [v2:127.0.0.1:6810/589093976,v1:127.0.0.1:6811/589093976] [v2:127.0.0.1:6812/589093976,v1:127.0.0.1:6813/589093976] exists,up c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:06:51.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:06:51.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:06:51.838 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:06:51.839 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:06:51.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:06:51.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:06:51.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:06:51.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:06:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:06:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:06:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:06:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:06:52.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:52.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:06:52.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-06-11T18:06:52.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-06-11T18:06:52.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-06-11T18:06:52.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:52.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:06:52.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757250 2024-06-11T18:06:52.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757250 2024-06-11T18:06:52.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-163208757250' 2024-06-11T18:06:52.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:06:52.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:06:52.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-06-11T18:06:52.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-06-11T18:06:52.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-163208757250 2-64424509453' 2024-06-11T18:06:52.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:52.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:52.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-06-11T18:06:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:06:52.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-06-11T18:06:52.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:52.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-06-11T18:06:52.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-11T18:06:52.793 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836506 2024-06-11T18:06:52.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:06:53.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836506 2024-06-11T18:06:53.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:06:54.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:06:54.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:06:54.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836506 2024-06-11T18:06:54.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:54.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:54.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-163208757250 2024-06-11T18:06:54.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:06:54.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:54.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-163208757250 2024-06-11T18:06:54.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757250 2024-06-11T18:06:54.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-06-11T18:06:54.466 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 163208757250 2024-06-11T18:06:54.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:06:54.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757250 -lt 163208757250 2024-06-11T18:06:54.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:06:54.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-06-11T18:06:54.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:06:54.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:06:54.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-06-11T18:06:54.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:06:54.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-06-11T18:06:54.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-11T18:06:54.800 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509453 2024-06-11T18:06:54.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:06:55.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-11T18:06:55.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:06:55.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:55.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:55.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:06:55.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:06:55.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:06:55.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:06:55.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:06:55.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:06:55.561 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:06:55.561 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:06:55.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:06:55.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:06:55.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:06:55.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-11T18:06:56.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:06:56.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:06:56.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-11T18:06:56.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:06:56.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:06:56.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:06:56.733 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:06:56.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:06:57.261 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:06:57.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:06:57.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:06:57.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:06:57.476 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-11T18:06:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:06:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:06:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:1+0 records in 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:1+0 records out 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000167159 s, 12.3 MB/s 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:06:57.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:06:57.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:06:57.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:06:57.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:06:57.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:06:57.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:06:57.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:06:57.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:06:57.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:06:57.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:06:57.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:07:00.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:07:00.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:07:00.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:07:00.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:07:00.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:07:00.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:07:00.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:07:00.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:07:00.300 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:07:00.300 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:07:00.300 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:07:00.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:00.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:07:00.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:07:00.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:07:00.302 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:07:00.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:07:00.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:07:00.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:07:00.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:07:00.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:07:00.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:07:00.342 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:00.339+0000 7f536d5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:00.345 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:00.339+0000 7f536d5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:00.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:00.343+0000 7f536d5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:00.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:07:00.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:00.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:00.897 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:00.891+0000 7f536d5777c0 -1 Falling back to public interface 2024-06-11T18:07:00.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:07:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:01.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:01.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:02.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:03.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:03.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:03.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:07:03.310 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:07:03.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:03.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:03.649 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:03.643+0000 7f536d5777c0 -1 osd.0 42 log_to_monitors true 2024-06-11T18:07:03.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:04.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:04.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:04.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:07:04.765 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:07:04.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:04.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:05.202 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 43 last_clean_interval [5,42) [v2:127.0.0.1:6802/1360248472,v1:127.0.0.1:6803/1360248472] [v2:127.0.0.1:6804/1360248472,v1:127.0.0.1:6805/1360248472] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:07:05.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:07:05.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:07:05.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:07:05.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:05.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:07:05.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462914 2024-06-11T18:07:05.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462914 2024-06-11T18:07:05.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914' 2024-06-11T18:07:05.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:05.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:07:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757254 2024-06-11T18:07:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757254 2024-06-11T18:07:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914 1-163208757254' 2024-06-11T18:07:05.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:05.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:07:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-06-11T18:07:06.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-06-11T18:07:06.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914 1-163208757254 2-64424509457' 2024-06-11T18:07:06.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:06.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-201863462914 2024-06-11T18:07:06.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:06.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:07:06.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-201863462914 2024-06-11T18:07:06.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462914 2024-06-11T18:07:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2024-06-11T18:07:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 201863462914 2024-06-11T18:07:06.123 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:06.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 201863462914 2024-06-11T18:07:06.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:07:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:07:07.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:07.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462914 -lt 201863462914 2024-06-11T18:07:07.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:07.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-163208757254 2024-06-11T18:07:07.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:07.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:07:07.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-163208757254 2024-06-11T18:07:07.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:07.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757254 2024-06-11T18:07:07.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 163208757254' 2024-06-11T18:07:07.769 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 163208757254 2024-06-11T18:07:07.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757254 -lt 163208757254 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:07:08.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-06-11T18:07:08.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:08.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-06-11T18:07:08.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-11T18:07:08.112 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509457 2024-06-11T18:07:08.112 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:07:08.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-11T18:07:08.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:07:08.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:08.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:08.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:07:08.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:07:08.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:07:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:07:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:07:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:07:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:07:08.839 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:07:09.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:07:09.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:07:09.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:09.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:09.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:07:09.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:07:09.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:07:09.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:07:09.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:07:09.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:07:09.996 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:07:10.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:07:10.523 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:07:10.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:07:10.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:07:10.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:07:10.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:07:10.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:07:10.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:07:11.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:07:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-11T18:07:12.745 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:07:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:07:13.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:07:13.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:07:13.289 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:07:13.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:07:13.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:07:13.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:07:13.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:07:13.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:07:13.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:07:13.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:13.327+0000 7f18ade8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:13.356 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:13.351+0000 7f18ade8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:13.357 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:13.351+0000 7f18ade8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:07:13.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:13.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:07:13.629 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:07:13.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:13.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:07:13.892 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:13.887+0000 7f18ade8e7c0 -1 Falling back to public interface 2024-06-11T18:07:13.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:14.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:14.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:14.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:07:14.967 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:14.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:14.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:07:15.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:16.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:16.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:16.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:07:16.296 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:07:16.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:16.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:07:16.336 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:16.331+0000 7f18ade8e7c0 -1 osd.1 50 log_to_monitors true 2024-06-11T18:07:16.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:17.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:17.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:17.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:07:17.691 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:07:17.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:17.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:07:18.134 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [38,50) [v2:127.0.0.1:6810/3127188696,v1:127.0.0.1:6811/3127188696] [v2:127.0.0.1:6812/3127188696,v1:127.0.0.1:6813/3127188696] exists,up c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:07:18.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:07:18.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:07:18.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:07:18.137 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:07:18.137 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:07:18.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:07:18.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:07:18.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:07:18.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:07:18.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:07:18.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:07:18.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:07:18.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:07:18.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:07:18.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:07:18.588 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:18.588 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:07:18.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:07:18.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:18.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:07:18.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462917 2024-06-11T18:07:18.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462917 2024-06-11T18:07:18.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462917' 2024-06-11T18:07:18.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:18.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:07:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=236223201282 2024-06-11T18:07:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 236223201282 2024-06-11T18:07:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462917 1-236223201282' 2024-06-11T18:07:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:18.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:07:19.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-06-11T18:07:19.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-06-11T18:07:19.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462917 1-236223201282 2-64424509460' 2024-06-11T18:07:19.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:19.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-201863462917 2024-06-11T18:07:19.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:19.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:07:19.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-201863462917 2024-06-11T18:07:19.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:19.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462917 2024-06-11T18:07:19.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 201863462917' 2024-06-11T18:07:19.053 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 201863462917 2024-06-11T18:07:19.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:19.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462918 -lt 201863462917 2024-06-11T18:07:19.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:19.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-236223201282 2024-06-11T18:07:19.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:19.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:07:19.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-236223201282 2024-06-11T18:07:19.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:19.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=236223201282 2024-06-11T18:07:19.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-06-11T18:07:19.386 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 236223201282 2024-06-11T18:07:19.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:07:19.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 236223201282 -lt 236223201282 2024-06-11T18:07:19.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:19.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-06-11T18:07:19.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:19.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:07:19.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-06-11T18:07:19.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:19.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-06-11T18:07:19.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-11T18:07:19.728 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509460 2024-06-11T18:07:19.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:07:20.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-11T18:07:20.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:07:20.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:20.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:20.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:07:20.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:07:20.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:07:20.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:07:20.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:07:20.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:20.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:07:21.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-11T18:07:21.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:07:21.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:07:21.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:07:21.674 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:07:21.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:07:22.117 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:07:22.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:07:22.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:07:22.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:07:22.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-11T18:07:22.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:07:22.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:07:22.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-11T18:07:22.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-11T18:07:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-11T18:07:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:07:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:07:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:07:22.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:07:22.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:07:22.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:07:22.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:07:22.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:07:23.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:07:23.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:07:23.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:07:23.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:07:23.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:07:23.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:07:23.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:07:23.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:07:25.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:07:25.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:07:25.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:07:25.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:07:25.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:07:25.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:07:25.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:07:25.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:07:25.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:07:25.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:07:25.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:07:25.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:07:25.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:07:25.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:25.463+0000 7faa190ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:25.467+0000 7faa190ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:25.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:25.471+0000 7faa190ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:07:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:07:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:07:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:07:25.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:07:25.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:25.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:07:25.770 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:07:25.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:25.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:26.020 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:26.015+0000 7faa190ec7c0 -1 Falling back to public interface 2024-06-11T18:07:26.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:27.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:27.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:27.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:07:27.120 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:27.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:27.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:27.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:28.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:28.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:28.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:07:28.455 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:07:28.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:28.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:28.496 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:28.491+0000 7faa190ec7c0 -1 osd.0 59 log_to_monitors true 2024-06-11T18:07:28.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:29.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:29.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:29.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:07:29.807 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:07:29.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:29.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:07:30.156 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [47,59) [v2:127.0.0.1:6802/384667049,v1:127.0.0.1:6803/384667049] [v2:127.0.0.1:6804/384667049,v1:127.0.0.1:6805/384667049] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:07:30.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:07:30.158 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:07:30.158 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:07:30.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:07:30.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:07:30.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:07:30.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:07:30.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:07:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:07:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:07:30.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:07:30.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:07:30.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:07:30.608 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:30.608 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:07:30.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:07:30.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:30.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:07:30.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906946 2024-06-11T18:07:30.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906946 2024-06-11T18:07:30.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906946' 2024-06-11T18:07:30.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:30.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:07:30.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=236223201285 2024-06-11T18:07:30.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 236223201285 2024-06-11T18:07:30.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906946 1-236223201285' 2024-06-11T18:07:30.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:30.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:07:31.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509464 2024-06-11T18:07:31.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509464 2024-06-11T18:07:31.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906946 1-236223201285 2-64424509464' 2024-06-11T18:07:31.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:31.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:31.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-274877906946 2024-06-11T18:07:31.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:07:31.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-274877906946 2024-06-11T18:07:31.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:31.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906946 2024-06-11T18:07:31.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 274877906946' 2024-06-11T18:07:31.072 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 274877906946 2024-06-11T18:07:31.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:31.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906946 -lt 274877906946 2024-06-11T18:07:31.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:31.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-236223201285 2024-06-11T18:07:31.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:31.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:07:31.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-236223201285 2024-06-11T18:07:31.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:31.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=236223201285 2024-06-11T18:07:31.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 236223201285' 2024-06-11T18:07:31.401 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 236223201285 2024-06-11T18:07:31.402 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:07:31.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 236223201285 -lt 236223201285 2024-06-11T18:07:31.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:31.725 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509464 2024-06-11T18:07:31.725 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:31.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:07:31.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509464 2024-06-11T18:07:31.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:31.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509464 2024-06-11T18:07:31.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-11T18:07:31.729 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 64424509464 2024-06-11T18:07:31.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:07:32.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-11T18:07:32.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:07:32.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:32.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:32.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:07:32.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:07:32.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:07:32.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:07:32.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:07:32.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:07:32.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:07:32.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:07:32.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:07:32.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:07:32.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:32.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:33.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:07:33.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:07:33.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:07:33.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:07:33.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:07:33.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:07:33.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:07:35.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T18:07:35.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:07:35.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:07:35.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:07:35.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:07:35.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:07:35.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:07:35.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:07:35.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:07:35.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:07:35.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:07:35.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:07:35.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:07:35.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:07:35.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:07:35.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:07:35.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:07:35.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:35.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:07:35.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:07:35.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:07:35.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:07:35.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:07:35.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:07:35.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:07:35.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:07:35.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:07:35.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:07:35.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:07:35.685 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:07:35.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:07:35.685 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:07:35.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:07:35.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:07:35.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:07:35.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:07:35.714 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:35.707+0000 7f3bb7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:35.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:35.711+0000 7f3bb7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:35.718 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:35.715+0000 7f3bb7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:07:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:07:36.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:36.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:36.260 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:36.255+0000 7f3bb7a6a7c0 -1 Falling back to public interface 2024-06-11T18:07:36.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:37.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:37.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:37.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:07:37.356 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:37.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:37.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:37.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:38.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:38.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:38.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:07:38.698 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:07:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:38.793 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:38.787+0000 7f3bb7a6a7c0 -1 osd.2 66 log_to_monitors true 2024-06-11T18:07:39.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:40.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:40.579 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 71 up_thru 71 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/2969164436,v1:127.0.0.1:6819/2969164436] [v2:127.0.0.1:6820/2969164436,v1:127.0.0.1:6821/2969164436] exists,up bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:07:40.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:07:40.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:07:40.581 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:07:40.581 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:07:40.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:07:40.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:07:40.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:07:40.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:07:40.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:07:40.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:07:40.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:07:40.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:07:40.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:07:41.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:07:41.036 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:41.036 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:07:41.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:07:41.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:41.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:07:41.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906949 2024-06-11T18:07:41.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906949 2024-06-11T18:07:41.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906949' 2024-06-11T18:07:41.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:41.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:07:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=236223201288 2024-06-11T18:07:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 236223201288 2024-06-11T18:07:41.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906949 1-236223201288' 2024-06-11T18:07:41.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:41.353 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:07:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=304942678018 2024-06-11T18:07:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 304942678018 2024-06-11T18:07:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906949 1-236223201288 2-304942678018' 2024-06-11T18:07:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:41.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-274877906949 2024-06-11T18:07:41.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:41.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:07:41.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-274877906949 2024-06-11T18:07:41.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:41.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906949 2024-06-11T18:07:41.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 274877906949' 2024-06-11T18:07:41.505 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 274877906949 2024-06-11T18:07:41.506 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:41.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906948 -lt 274877906949 2024-06-11T18:07:41.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:07:42.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:07:42.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:43.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906948 -lt 274877906949 2024-06-11T18:07:43.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:07:44.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:07:44.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:44.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906949 -lt 274877906949 2024-06-11T18:07:44.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:44.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-236223201288 2024-06-11T18:07:44.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:44.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:07:44.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:44.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-236223201288 2024-06-11T18:07:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=236223201288 2024-06-11T18:07:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 236223201288' 2024-06-11T18:07:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 236223201288 2024-06-11T18:07:44.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:07:44.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 236223201289 -lt 236223201288 2024-06-11T18:07:44.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:44.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-304942678018 2024-06-11T18:07:44.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:44.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:07:44.870 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-304942678018 2024-06-11T18:07:44.870 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=304942678018 2024-06-11T18:07:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-06-11T18:07:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 304942678018 2024-06-11T18:07:44.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:07:45.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 304942678018 -lt 304942678018 2024-06-11T18:07:45.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:07:45.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:45.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:45.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:07:45.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:07:45.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:07:45.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:07:45.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:07:45.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:45.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T18:07:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:07:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:07:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:07:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:07:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:07:46.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:07:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T18:07:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:07:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:07:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:07:48.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:07:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:07:48.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:07:48.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:07:48.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:07:48.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:07:48.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:07:48.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:07:48.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:07:48.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:07:48.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:07:48.696 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:07:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:07:48.697 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:07:48.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:07:48.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:07:48.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:07:48.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:07:48.731 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:48.727+0000 7f87f9ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:48.733 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:48.727+0000 7f87f9ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:48.735 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:48.731+0000 7f87f9ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:07:49.045 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:07:49.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:49.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:49.269 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:49.263+0000 7f87f9ec57c0 -1 Falling back to public interface 2024-06-11T18:07:49.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:50.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:50.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:50.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:07:50.422 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:50.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:50.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:50.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:51.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:51.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:51.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:07:51.756 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:07:51.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:51.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:51.788 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:07:51.783+0000 7f87f9ec57c0 -1 osd.2 73 log_to_monitors true 2024-06-11T18:07:52.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:07:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:07:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:07:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:07:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:07:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:07:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 78 up_thru 78 down_at 74 last_clean_interval [71,73) [v2:127.0.0.1:6818/4005254305,v1:127.0.0.1:6819/4005254305] [v2:127.0.0.1:6820/4005254305,v1:127.0.0.1:6821/4005254305] exists,up bbd7068f-67e0-41de-be9f-c78dd0d005e4 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:07:53.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:07:53.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:07:53.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:07:53.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:07:53.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:07:53.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:07:53.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:07:53.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:07:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:07:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:07:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:07:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:07:53.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:07:54.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:07:54.064 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:07:54.064 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:07:54.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:07:54.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:54.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:07:54.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906953 2024-06-11T18:07:54.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906953 2024-06-11T18:07:54.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906953' 2024-06-11T18:07:54.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:54.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:07:54.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=236223201292 2024-06-11T18:07:54.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 236223201292 2024-06-11T18:07:54.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906953 1-236223201292' 2024-06-11T18:07:54.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:07:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:07:54.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449090 2024-06-11T18:07:54.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449090 2024-06-11T18:07:54.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906953 1-236223201292 2-335007449090' 2024-06-11T18:07:54.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:54.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-274877906953 2024-06-11T18:07:54.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:54.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:07:54.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:54.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-274877906953 2024-06-11T18:07:54.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906953 2024-06-11T18:07:54.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 274877906953' 2024-06-11T18:07:54.526 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 274877906953 2024-06-11T18:07:54.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:54.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906951 -lt 274877906953 2024-06-11T18:07:54.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:07:55.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:07:55.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:07:56.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906953 -lt 274877906953 2024-06-11T18:07:56.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:56.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:56.192 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-236223201292 2024-06-11T18:07:56.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:07:56.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-236223201292 2024-06-11T18:07:56.194 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:56.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=236223201292 2024-06-11T18:07:56.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 236223201292' 2024-06-11T18:07:56.196 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 236223201292 2024-06-11T18:07:56.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:07:56.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 236223201292 -lt 236223201292 2024-06-11T18:07:56.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:07:56.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:07:56.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449090 2024-06-11T18:07:56.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:07:56.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449090 2024-06-11T18:07:56.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:07:56.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449090 2024-06-11T18:07:56.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-06-11T18:07:56.533 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449090 2024-06-11T18:07:56.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:07:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449090 -lt 335007449090 2024-06-11T18:07:56.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:07:56.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:56.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:57.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:07:57.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:07:57.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:07:57.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:07:57.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:07:57.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:07:57.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:07:57.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:07:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:07:57.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:07:57.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:07:57.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:07:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:07:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:07:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:07:58.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-11T18:07:58.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:07:58.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:07:58.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:07:58.522 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:07:58.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:07:58.973 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:07:58.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:07:58.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:07:58.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:07:59.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-11T18:07:59.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:07:59.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:07:59.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-11T18:07:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:2+0 records in 2024-06-11T18:07:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:2+0 records out 2024-06-11T18:07:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000207189 s, 19.8 MB/s 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:07:59.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:07:59.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:07:59.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:07:59.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:07:59.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:07:59.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:08:00.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:08:02.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:08:02.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:08:02.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:08:02.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:08:02.478 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:08:02.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:02.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:08:02.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:08:02.480 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:08:02.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:08:02.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:08:02.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:08:02.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:08:02.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:08:02.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:08:02.520 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:02.515+0000 7fd40e8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:02.523 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:02.519+0000 7fd40e8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:02.525 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:02.519+0000 7fd40e8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:08:02.818 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:08:02.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:02.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:03.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:03.047+0000 7fd40e8807c0 -1 Falling back to public interface 2024-06-11T18:08:03.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:04.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:04.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:04.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:08:04.155 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:04.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:04.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:04.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:05.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:05.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:05.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:08:05.483 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:08:05.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:05.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:05.739 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:05.735+0000 7fd40e8807c0 -1 osd.1 82 log_to_monitors true 2024-06-11T18:08:05.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:06.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:06.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:06.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:08:06.874 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:08:06.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:06.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 83 last_clean_interval [55,82) [v2:127.0.0.1:6810/1062150525,v1:127.0.0.1:6811/1062150525] [v2:127.0.0.1:6812/1062150525,v1:127.0.0.1:6813/1062150525] exists,up c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:08:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:08:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:08:07.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:08:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:08:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:08:07.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:08:07.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:08:07.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:08:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:08:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:08:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:08:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:08:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:08:07.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:08:07.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:08:07.761 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:07.761 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:08:07.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:08:07.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:07.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:08:07.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=274877906956 2024-06-11T18:08:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 274877906956 2024-06-11T18:08:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906956' 2024-06-11T18:08:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:07.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:08:08.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=373662154754 2024-06-11T18:08:08.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 373662154754 2024-06-11T18:08:08.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906956 1-373662154754' 2024-06-11T18:08:08.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:08.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:08:08.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449094 2024-06-11T18:08:08.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449094 2024-06-11T18:08:08.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-274877906956 1-373662154754 2-335007449094' 2024-06-11T18:08:08.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:08.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-274877906956 2024-06-11T18:08:08.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:08.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:08:08.292 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-274877906956 2024-06-11T18:08:08.292 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:08.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=274877906956 2024-06-11T18:08:08.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 274877906956' 2024-06-11T18:08:08.293 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 274877906956 2024-06-11T18:08:08.293 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:08.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906955 -lt 274877906956 2024-06-11T18:08:08.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:08:09.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:08:09.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:09.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 274877906957 -lt 274877906956 2024-06-11T18:08:09.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:09.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-373662154754 2024-06-11T18:08:09.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:09.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:08:09.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-373662154754 2024-06-11T18:08:09.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:09.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=373662154754 2024-06-11T18:08:09.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-06-11T18:08:09.961 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 373662154754 2024-06-11T18:08:09.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:10.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 373662154754 -lt 373662154754 2024-06-11T18:08:10.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:10.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:10.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449094 2024-06-11T18:08:10.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:08:10.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449094 2024-06-11T18:08:10.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:10.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449094 2024-06-11T18:08:10.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449094' 2024-06-11T18:08:10.307 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449094 2024-06-11T18:08:10.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:08:10.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449094 -lt 335007449094 2024-06-11T18:08:10.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:08:10.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:10.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:11.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:08:11.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:08:11.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:08:11.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:08:11.399 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:08:11.399 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:11.399 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:11.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:08:11.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:08:11.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:08:11.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:08:11.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:08:12.219 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:08:12.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:08:12.724 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:08:12.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:08:12.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:08:12.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:08:12.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-11T18:08:12.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:08:12.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:08:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:08:12.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:08:12.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:08:12.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:08:12.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:08:12.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:08:13.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:13.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:08:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:08:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:08:15.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:08:15.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:08:15.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:08:15.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:08:15.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:08:15.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:08:15.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:08:15.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:08:15.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:08:15.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:08:15.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:08:15.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:08:15.493 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:15.487+0000 7f2a7090e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:15.496 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:15.491+0000 7f2a7090e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:15.497 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:15.491+0000 7f2a7090e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:08:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:15.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:16.033 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:16.027+0000 7f2a7090e7c0 -1 Falling back to public interface 2024-06-11T18:08:16.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:17.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:17.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:17.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:08:17.123 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:17.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:17.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:17.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:18.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:18.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:18.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:08:18.460 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:08:18.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:18.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:18.635 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:18.631+0000 7f2a7090e7c0 -1 osd.0 91 log_to_monitors true 2024-06-11T18:08:18.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:19.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:19.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:19.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:08:19.925 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:08:19.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:19.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:20.264 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [64,91) [v2:127.0.0.1:6802/433850177,v1:127.0.0.1:6803/433850177] [v2:127.0.0.1:6804/433850177,v1:127.0.0.1:6805/433850177] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:08:20.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:08:20.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:08:20.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:08:20.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:20.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:08:20.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860418 2024-06-11T18:08:20.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860418 2024-06-11T18:08:20.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860418' 2024-06-11T18:08:20.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:20.879 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:08:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=373662154757 2024-06-11T18:08:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 373662154757 2024-06-11T18:08:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860418 1-373662154757' 2024-06-11T18:08:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:21.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:08:21.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449097 2024-06-11T18:08:21.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449097 2024-06-11T18:08:21.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860418 1-373662154757 2-335007449097' 2024-06-11T18:08:21.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:21.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:21.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-412316860418 2024-06-11T18:08:21.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:08:21.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-412316860418 2024-06-11T18:08:21.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:21.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860418 2024-06-11T18:08:21.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2024-06-11T18:08:21.172 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 412316860418 2024-06-11T18:08:21.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860418 -lt 412316860418 2024-06-11T18:08:21.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:21.505 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-373662154757 2024-06-11T18:08:21.505 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:21.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:08:21.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-373662154757 2024-06-11T18:08:21.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:21.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=373662154757 2024-06-11T18:08:21.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 373662154757' 2024-06-11T18:08:21.510 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 373662154757 2024-06-11T18:08:21.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:21.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 373662154758 -lt 373662154757 2024-06-11T18:08:21.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:21.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449097 2024-06-11T18:08:21.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:21.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:08:21.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449097 2024-06-11T18:08:21.864 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:21.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449097 2024-06-11T18:08:21.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449097' 2024-06-11T18:08:21.866 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449097 2024-06-11T18:08:21.866 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:08:22.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449097 -lt 335007449097 2024-06-11T18:08:22.200 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:08:22.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:22.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:22.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:08:22.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:08:22.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:08:22.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:08:22.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:08:22.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:08:22.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:08:22.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:08:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:08:22.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:08:22.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:22.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:08:23.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:08:23.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:08:23.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:08:23.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:08:23.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:08:23.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:08:25.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:08:25.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:08:25.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:08:25.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:08:25.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:08:25.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:08:25.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:08:25.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:08:25.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:08:25.680 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:25.675+0000 7f164983b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:25.683 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:25.679+0000 7f164983b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:25.685 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:25.679+0000 7f164983b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:25.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:26.225 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:26.219+0000 7f164983b7c0 -1 Falling back to public interface 2024-06-11T18:08:26.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:27.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:27.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:28.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:28.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:28.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:08:28.651 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:08:28.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:28.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:28.737 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:28.731+0000 7f164983b7c0 -1 osd.1 98 log_to_monitors true 2024-06-11T18:08:29.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:30.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:30.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:30.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:08:30.057 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:08:30.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:30.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:08:30.444 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 103 up_thru 103 down_at 99 last_clean_interval [87,98) [v2:127.0.0.1:6810/4015492659,v1:127.0.0.1:6811/4015492659] [v2:127.0.0.1:6812/4015492659,v1:127.0.0.1:6813/4015492659] exists,up c0fb9163-8a9f-480a-839a-b8ff3904b239 2024-06-11T18:08:30.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:08:30.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:08:30.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:08:30.446 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:08:30.446 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:08:30.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:08:30.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:08:30.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:08:30.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:08:30.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:08:30.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:08:30.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:08:30.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:08:30.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:30.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:08:31.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=412316860421 2024-06-11T18:08:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 412316860421 2024-06-11T18:08:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860421' 2024-06-11T18:08:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:31.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:08:31.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=442381631490 2024-06-11T18:08:31.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 442381631490 2024-06-11T18:08:31.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860421 1-442381631490' 2024-06-11T18:08:31.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:31.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:08:31.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449100 2024-06-11T18:08:31.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449100 2024-06-11T18:08:31.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-412316860421 1-442381631490 2-335007449100' 2024-06-11T18:08:31.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:31.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-412316860421 2024-06-11T18:08:31.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:31.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:08:31.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-412316860421 2024-06-11T18:08:31.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:31.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=412316860421 2024-06-11T18:08:31.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 412316860421' 2024-06-11T18:08:31.395 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 412316860421 2024-06-11T18:08:31.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:31.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 412316860421 -lt 412316860421 2024-06-11T18:08:31.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:31.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:31.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-442381631490 2024-06-11T18:08:31.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:08:31.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-442381631490 2024-06-11T18:08:31.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:31.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=442381631490 2024-06-11T18:08:31.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 442381631490' 2024-06-11T18:08:31.725 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 442381631490 2024-06-11T18:08:31.725 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:32.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 442381631490 2024-06-11T18:08:32.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:08:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:08:33.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:33.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 442381631490 -lt 442381631490 2024-06-11T18:08:33.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:33.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449100 2024-06-11T18:08:33.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:33.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:08:33.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449100 2024-06-11T18:08:33.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:33.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449100 2024-06-11T18:08:33.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449100' 2024-06-11T18:08:33.393 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449100 2024-06-11T18:08:33.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:08:33.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449101 -lt 335007449100 2024-06-11T18:08:33.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:08:33.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:33.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:34.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:08:34.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:08:34.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:08:34.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:08:34.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:08:34.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:08:34.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:08:34.169 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:08:34.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:08:34.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:08:34.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:34.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:08:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:08:34.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:08:34.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:08:35.495 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:08:35.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:08:35.946 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:08:35.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:08:35.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:08:35.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:08:36.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-11T18:08:36.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:08:36.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:08:36.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-11T18:08:36.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-11T18:08:36.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:08:36.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:08:36.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:08:36.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:08:36.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:08:36.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:36.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-11T18:08:38.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:08:38.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:08:38.739 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:08:38.740 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:08:38.740 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:08:38.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:08:38.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:08:38.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:08:38.742 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:08:38.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:08:38.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:08:38.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:08:38.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:08:38.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:08:38.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:08:38.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:38.771+0000 7fe6e14eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:38.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:38.775+0000 7fe6e14eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:38.775+0000 7fe6e14eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:39.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:08:39.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:39.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:39.324 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:39.319+0000 7fe6e14eb7c0 -1 Falling back to public interface 2024-06-11T18:08:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:40.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:40.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:40.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:08:40.449 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:40.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:40.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:40.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:41.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:08:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:08:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:41.841 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:41.835+0000 7fe6e14eb7c0 -1 osd.0 107 log_to_monitors true 2024-06-11T18:08:42.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:43.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:43.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:43.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:08:43.157 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:08:43.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:43.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:43.659 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 112 up_thru 112 down_at 108 last_clean_interval [96,107) [v2:127.0.0.1:6802/2458221724,v1:127.0.0.1:6803/2458221724] [v2:127.0.0.1:6804/2458221724,v1:127.0.0.1:6805/2458221724] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:08:43.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:08:43.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:08:43.660 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:08:43.661 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:08:43.661 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:08:43.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:08:43.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:08:43.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:08:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:08:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:08:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:08:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:08:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:08:43.791 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:08:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:08:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:08:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:08:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:44.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:08:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=481036337154 2024-06-11T18:08:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 481036337154 2024-06-11T18:08:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-481036337154' 2024-06-11T18:08:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:44.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:08:44.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=442381631494 2024-06-11T18:08:44.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 442381631494 2024-06-11T18:08:44.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-481036337154 1-442381631494' 2024-06-11T18:08:44.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:44.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:08:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449104 2024-06-11T18:08:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449104 2024-06-11T18:08:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-481036337154 1-442381631494 2-335007449104' 2024-06-11T18:08:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:44.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-481036337154 2024-06-11T18:08:44.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:44.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:08:44.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-481036337154 2024-06-11T18:08:44.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=481036337154 2024-06-11T18:08:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 481036337154' 2024-06-11T18:08:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 481036337154 2024-06-11T18:08:44.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 481036337154 2024-06-11T18:08:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:08:45.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:08:45.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:46.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 481036337154 -lt 481036337154 2024-06-11T18:08:46.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:46.243 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-442381631494 2024-06-11T18:08:46.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:46.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:08:46.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-442381631494 2024-06-11T18:08:46.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=442381631494 2024-06-11T18:08:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 442381631494' 2024-06-11T18:08:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 442381631494 2024-06-11T18:08:46.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:46.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 442381631494 -lt 442381631494 2024-06-11T18:08:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:46.571 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449104 2024-06-11T18:08:46.571 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:46.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:08:46.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449104 2024-06-11T18:08:46.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:46.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449104 2024-06-11T18:08:46.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449104' 2024-06-11T18:08:46.575 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449104 2024-06-11T18:08:46.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:08:46.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449104 -lt 335007449104 2024-06-11T18:08:46.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:08:46.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:46.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:47.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:08:47.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:08:47.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:08:47.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:08:47.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:08:47.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:08:47.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:08:47.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:08:47.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:08:47.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:08:47.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:47.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:08:48.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:08:48.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:08:48.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:08:48.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:08:48.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:08:48.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:08:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:08:50.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:08:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:08:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:08:50.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:08:50.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:08:50.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:08:50.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:08:50.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:08:50.925 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:50.918+0000 7f4280eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:50.927 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:50.922+0000 7f4280eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:50.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:50.922+0000 7f4280eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:08:51.225 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:08:51.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:51.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:51.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:51.470+0000 7f4280eb47c0 -1 Falling back to public interface 2024-06-11T18:08:51.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:52.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:52.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:52.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:08:52.598 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:52.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:52.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:52.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:53.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:53.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:53.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:08:53.931 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:08:53.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:53.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:53.962 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:08:53.958+0000 7f4280eb47c0 -1 osd.0 114 log_to_monitors true 2024-06-11T18:08:54.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:08:55.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:08:55.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:08:55.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:08:55.321 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:08:55.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:08:55.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:08:55.731 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 119 up_thru 119 down_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6802/3998073748,v1:127.0.0.1:6803/3998073748] [v2:127.0.0.1:6804/3998073748,v1:127.0.0.1:6805/3998073748] exists,up f2ea95ed-6d17-4c91-9547-90a369b22c6e 2024-06-11T18:08:55.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:08:55.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:08:55.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:08:55.733 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:08:55.733 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:08:55.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:08:55.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:08:55.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:08:55.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:08:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:08:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:08:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:08:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:08:56.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:56.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:08:56.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=511101108226 2024-06-11T18:08:56.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 511101108226 2024-06-11T18:08:56.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-511101108226' 2024-06-11T18:08:56.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:56.348 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:08:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=442381631497 2024-06-11T18:08:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 442381631497 2024-06-11T18:08:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-511101108226 1-442381631497' 2024-06-11T18:08:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:08:56.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:08:56.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=335007449107 2024-06-11T18:08:56.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 335007449107 2024-06-11T18:08:56.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-511101108226 1-442381631497 2-335007449107' 2024-06-11T18:08:56.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:56.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-511101108226 2024-06-11T18:08:56.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:56.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:08:56.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-511101108226 2024-06-11T18:08:56.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:56.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=511101108226 2024-06-11T18:08:56.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 511101108226' 2024-06-11T18:08:56.646 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 511101108226 2024-06-11T18:08:56.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 511101108226 2024-06-11T18:08:56.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:08:57.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:08:57.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:08:58.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 511101108226 -lt 511101108226 2024-06-11T18:08:58.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:58.294 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-442381631497 2024-06-11T18:08:58.294 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:58.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:08:58.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-442381631497 2024-06-11T18:08:58.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:58.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=442381631497 2024-06-11T18:08:58.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 442381631497' 2024-06-11T18:08:58.298 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 442381631497 2024-06-11T18:08:58.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:08:58.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 442381631497 -lt 442381631497 2024-06-11T18:08:58.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:08:58.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-335007449107 2024-06-11T18:08:58.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:08:58.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:08:58.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-335007449107 2024-06-11T18:08:58.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:08:58.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=335007449107 2024-06-11T18:08:58.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 335007449107' 2024-06-11T18:08:58.645 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 335007449107 2024-06-11T18:08:58.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:08:58.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 335007449108 -lt 335007449107 2024-06-11T18:08:58.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:08:58.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:58.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:08:59.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:08:59.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:08:59.395 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:08:59.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:08:59.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:08:59.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:08:59.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:09:00.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:09:00.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:09:00.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:09:00.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-11T18:09:00.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-11T18:09:00.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:09:00.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-11T18:09:00.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:09:00.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-11T18:09:00.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-11T18:09:00.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-11T18:09:00.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=3.0 2024-06-11T18:09:00.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 3.0 2024-06-11T18:09:00.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:00.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:09:00.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:00.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:09:00.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:00.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 3.0 2024-06-11T18:09:01.113 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 3.0 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:09:01.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:01.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:09:01.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:01.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:09:01.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:01.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:09:01.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:06:32.741682+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:01.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:09:02.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:09:02.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:02.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:09:02.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:02.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:09:02.475 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:02.475 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:09:02.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:06:32.741682+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:02.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:09:03.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:09:03.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:03.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:09:03.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:03.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:09:03.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:03.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:09:04.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:06:32.741682+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:04.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:09:05.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:09:05.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:05.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:09:05.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:05.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:09:05.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:05.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:09:05.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:09:01.902859+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:05.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:09:05.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-11T18:09:05.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:09:05.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-11T18:09:05.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:09:05.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-11T18:09:05.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-11T18:09:05.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:09:05.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=119 2024-06-11T18:09:05.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:09:05.613 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:09:05.613 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:09:05.613 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:09:05.613 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:09:05.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:09:05.653 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:09:05.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:09:05.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:09:05.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: local pgid=3.0 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:09:05.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:05.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:09:06.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: local last_scrub=2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:06.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-11T18:09:06.311 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-11T18:09:06.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: pg_deep_scrub: wait_for_scrub 3.0 2024-06-11T18:06:32.741682+0000 last_deep_scrub_stamp 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:06.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:09:06.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:06.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:09:06.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:09:06.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:06.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:06:32.741682+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:06.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:09:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:09:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:09:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:09:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:09:07.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:06:32.741682+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:07.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:09:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:09:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:09:08.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:09:08.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:09:08.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:09:08.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:09:08.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:09:09.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:09:06.997197+0000 '>' 2024-06-11T18:06:32.741682+0000 2024-06-11T18:09:09.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:09:09.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-11T18:09:09.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:09:09.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-11T18:09:09.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:09:09.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-11T18:09:09.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-11T18:09:09.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=119 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:09:09.440 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:09:09.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:09:09.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:09:09.482 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:09:09.482 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:09:09.482 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:09:09.482 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:09:09.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:09:09.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:09:09.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:09:09.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-11T18:09:09.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-11T18:09:09.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-11T18:09:09.942 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' removed 2024-06-11T18:09:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:09:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:09:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:09:09.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:09:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:09:09.967 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:09:09.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:09:09.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:09:09.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:09:10.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:09:10.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:09:10.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:09:10.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:09:10.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:09:10.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:09:10.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:09:10.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:09:10.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:09:10.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:09:10.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:09:10.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:09:10.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:09:10.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:09:10.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:09:10.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:10.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:10.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:09:10.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:09:10.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:09:10.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:09:10.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:09:10.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:09:10.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:09:10.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:09:10.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:09:10.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:09:10.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:09:10.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:09:10.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:09:10.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:09:10.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:09:10.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:09:10.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:09:10.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:09:10.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:09:10.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:09:10.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:09:10.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:10.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:10.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:09:10.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:09:10.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:09:10.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:09:10.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:09:10.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:10.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:10.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:09:10.350 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-11T18:09:10.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:09:10.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:09:10.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:09:10.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:09:10.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:09:10.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:09:10.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:09:10.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:09:10.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:09:10.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:09:10.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:10.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:10.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:10.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:09:11.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:09:11.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:09:11.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:09:11.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:09:11.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:09:11.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:09:11.041 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:09:11.041 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:09:11.041 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:09:11.042 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:09:11.042 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:11.042 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:11.042 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:09:11.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:09:11.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:09:11.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:09:11.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:09:11.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:09:11.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:09:11.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:09:11.160 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:09:11.160 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:09:11.160 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:09:11.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:09:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:12.106 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:12.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:12.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:09:12.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:09:12.143 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-11T18:09:12.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:09:12.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:09:12.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:12.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:12.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:12.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:09:12.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:09:12.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:09:12.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:09:12.149 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:09:12.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a9c778f-5dbc-4282-bc11-64a28c12f14f' 2024-06-11T18:09:12.150 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:09:12.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIkmhm4NHuCRAANOpjhqJtjlBzK7mGZcCW2A== 2024-06-11T18:09:12.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIkmhm4NHuCRAANOpjhqJtjlBzK7mGZcCW2A=="}' 2024-06-11T18:09:12.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a9c778f-5dbc-4282-bc11-64a28c12f14f -i td/osd-scrub-repair/0/new.json 2024-06-11T18:09:12.433 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:09:12.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:09:12.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIkmhm4NHuCRAANOpjhqJtjlBzK7mGZcCW2A== --osd-uuid 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:09:12.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.490+0000 7ff7d1d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.494+0000 7ff7d1d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:12.495 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.494+0000 7ff7d1d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:12.496 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.494+0000 7ff7d1d0a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:09:12.496 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.494+0000 7ff7d1d0a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:09:12.496 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.494+0000 7ff7d1d0a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:09:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:12.582+0000 7ff7d1d0a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:09:18.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:09:18.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:09:18.246 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:09:18.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:09:18.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:09:18.663 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:09:18.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:09:18.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:09:18.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:09:18.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:09:18.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:09:18.697 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:18.694+0000 7fcac39b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:18.732 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:18.730+0000 7fcac39b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:18.734 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:18.734+0000 7fcac39b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:09:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:09:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:09:19.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:19.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:19.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:19.278+0000 7fcac39b87c0 -1 Falling back to public interface 2024-06-11T18:09:19.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:20.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:20.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:20.364 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:09:20.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:09:20.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:20.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:20.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:21.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:21.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:21.695 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:09:21.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:09:21.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:21.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:21.789 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:21.786+0000 7fcac39b87c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:09:22.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:23.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:23.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:23.097 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:09:23.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:09:23.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:23.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:23.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:24.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:24.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:24.506 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:09:24.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:09:24.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:24.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:25.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:25.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:25.840 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:09:25.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:09:25.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:25.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:26.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:27.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:27.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:27.171 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:09:27.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:09:27.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:27.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:27.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:28.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:28.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:28.557 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:09:28.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:09:28.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:28.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:28.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:29.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:29.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:29.896 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:09:29.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:09:29.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:29.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:30.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:31.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:31.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:31.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:09:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:09:31.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:31.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:31.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:32.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:32.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:32.577 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:09:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:09:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:32.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:33.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:33.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:33.918 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:09:33.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:09:33.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:33.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:34.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:35.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:35.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:35.595 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:09:35.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:09:35.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:35.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:36.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:37.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:37.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:37.048 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:09:37.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:09:37.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:37.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:37.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:38.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:38.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:38.734 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:09:38.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:09:38.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:38.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:39.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:40.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:40.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:40.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:09:40.843 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:09:40.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:40.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:41.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:42.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:42.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:42.414 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:09:42.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:09:42.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:42.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:43.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:44.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:44.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:44.281 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:09:44.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:09:44.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:44.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:09:44.614 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4257104365,v1:127.0.0.1:6803/4257104365] [v2:127.0.0.1:6804/4257104365,v1:127.0.0.1:6805/4257104365] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:09:44.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:09:44.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:09:44.619 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:09:44.619 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:09:44.619 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:09:44.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:09:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:09:44.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:09:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:09:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 11c09347-73fb-4120-91f6-2d2a666028cb' 2024-06-11T18:09:44.621 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:09:44.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:09:44.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDokmhmm94cJhAAMl6dhsad9/5NaGP6LHAJ7A== 2024-06-11T18:09:44.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDokmhmm94cJhAAMl6dhsad9/5NaGP6LHAJ7A=="}' 2024-06-11T18:09:44.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11c09347-73fb-4120-91f6-2d2a666028cb -i td/osd-scrub-repair/1/new.json 2024-06-11T18:09:45.041 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:09:45.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:09:45.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDokmhmm94cJhAAMl6dhsad9/5NaGP6LHAJ7A== --osd-uuid 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:09:45.101 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.098+0000 7ffa926a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:45.126 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.126+0000 7ffa926a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:45.128 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.126+0000 7ffa926a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:45.129 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.126+0000 7ffa926a87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:09:45.129 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.126+0000 7ffa926a87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:09:45.129 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.126+0000 7ffa926a87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:09:45.199 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:45.198+0000 7ffa926a87c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:09:51.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:09:51.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:09:51.029 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:09:51.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:09:51.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:09:51.451 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:09:51.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:09:51.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:09:51.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:09:51.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:09:51.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:09:51.485 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:51.482+0000 7fe3bc2ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:51.531 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:51.530+0000 7fe3bc2ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:51.533 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:51.530+0000 7fe3bc2ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:51.804 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:09:51.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:09:51.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:51.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:52.080 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:52.078+0000 7fe3bc2ed7c0 -1 Falling back to public interface 2024-06-11T18:09:52.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:53.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:53.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:53.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:09:53.143 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:09:53.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:53.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:53.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:54.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:54.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:54.480 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:09:54.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:09:54.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:54.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:54.631 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:09:54.630+0000 7fe3bc2ed7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:09:54.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:55.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:55.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:55.860 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:09:55.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:09:55.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:55.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:56.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:57.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:57.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:57.202 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:09:57.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:09:57.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:57.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:58.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:58.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:58.601 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:09:58.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:09:58.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:58.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:09:58.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:09:59.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:09:59.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:09:59.941 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:09:59.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:09:59.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:09:59.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:00.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:01.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:01.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:01.313 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:10:01.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:10:01.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:01.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:01.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:02.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:02.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:02.654 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:10:02.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:10:02.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:02.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:02.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:03.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:03.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:03.988 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:10:03.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:10:03.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:03.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:04.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:05.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:05.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:05.314 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:10:05.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:10:05.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:06.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:06.655 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:10:06.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:06.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:10:06.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:06.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:06.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:07.985 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:10:07.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:10:07.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:07.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:09.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:09.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:09.650 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:10:09.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:10:09.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:09.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:10.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:11.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:11.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:11.260 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:10:11.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:10:11.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:11.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:12.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:12.792 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:10:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:10:12.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:12.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:13.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:14.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:14.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:14.175 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:10:14.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:10:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:14.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:15.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:16.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:16.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:16.091 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:10:16.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:10:16.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:16.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:16.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:17.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:17.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:17.662 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:10:17.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:10:17.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:17.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:10:18.052 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/405934203,v1:127.0.0.1:6811/405934203] [v2:127.0.0.1:6812/405934203,v1:127.0.0.1:6813/405934203] exists,up 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-11T18:10:18.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:10:18.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:10:18.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:10:18.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:10:18.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:10:18.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:10:18.062 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:10:18.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8477a069-9235-4ade-a413-40a5e44976a8' 2024-06-11T18:10:18.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:10:18.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKk2hmNdDUBBAAeowloLwjNePVNM5Wc+TAyw== 2024-06-11T18:10:18.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKk2hmNdDUBBAAeowloLwjNePVNM5Wc+TAyw=="}' 2024-06-11T18:10:18.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8477a069-9235-4ade-a413-40a5e44976a8 -i td/osd-scrub-repair/2/new.json 2024-06-11T18:10:18.490 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:10:18.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T18:10:18.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKk2hmNdDUBBAAeowloLwjNePVNM5Wc+TAyw== --osd-uuid 8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:10:18.549 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.546+0000 7f708df797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:18.552 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.550+0000 7f708df797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:18.553 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.550+0000 7f708df797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:18.553 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.550+0000 7f708df797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:10:18.553 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.550+0000 7f708df797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:10:18.554 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.550+0000 7f708df797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:10:18.632 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:18.630+0000 7f708df797c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T18:10:24.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T18:10:24.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:10:24.564 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T18:10:24.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T18:10:24.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:10:24.972 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T18:10:24.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T18:10:24.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:10:24.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:10:24.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:10:24.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:10:25.005 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:25.002+0000 7f0333e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:25.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:25.026+0000 7f0333e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:25.031 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:25.030+0000 7f0333e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:10:25.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T18:10:25.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:10:25.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:10:25.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:10:25.335 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:10:25.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:10:25.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:25.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:10:25.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:25.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:25.581 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:25.578+0000 7f0333e707c0 -1 Falling back to public interface 2024-06-11T18:10:25.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:26.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:26.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:26.676 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:10:26.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:10:26.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:26.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:27.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:28.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:28.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:28.010 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:10:28.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:10:28.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:28.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:28.141 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:10:28.138+0000 7f0333e707c0 -1 osd.2 0 log_to_monitors true 2024-06-11T18:10:28.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:29.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:29.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:29.347 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:10:29.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:10:29.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:29.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:29.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:30.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:30.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:30.797 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:10:30.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:10:30.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:30.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:31.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:32.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:32.154 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:10:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:10:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:32.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:32.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:33.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:33.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:33.480 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:10:33.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:10:33.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:33.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:33.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:34.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:34.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:34.976 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:10:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:10:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:35.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:36.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:36.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:36.306 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:10:36.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:10:36.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:36.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:36.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:37.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:37.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:37.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:10:37.635 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:10:37.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:37.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:37.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:38.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:38.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:38.967 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:10:38.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:10:38.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:38.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:39.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:40.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:40.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:40.308 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:10:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:10:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:40.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:41.647 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:10:41.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:10:41.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:41.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:42.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:43.364 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:10:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:10:43.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:43.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:44.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:45.298 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:10:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:10:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:45.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:46.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:46.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:46.662 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:10:46.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:10:46.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:46.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:47.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:48.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:48.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:48.391 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:10:48.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:10:48.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:48.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:48.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:49.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:49.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:49.916 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:10:49.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:10:49.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:49.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:50.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:10:51.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:10:51.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:10:51.595 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:10:51.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:10:51.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:10:51.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:10:51.931 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3756812905,v1:127.0.0.1:6819/3756812905] [v2:127.0.0.1:6820/3756812905,v1:127.0.0.1:6821/3756812905] exists,up 8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:10:51.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:10:51.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:10:51.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:10:51.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-11T18:10:51.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:10:52.341 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:10:52.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:10:52.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:10:52.816 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:10:52.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:10:53.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:10:55.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-11T18:10:55.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-11T18:10:56.075 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'foo' created 2024-06-11T18:10:56.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:10:57.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-06-11T18:10:57.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T18:10:57.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T18:10:57.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-06-11T18:10:57.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T18:10:57.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-11T18:10:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T18:10:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T18:10:58.526 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T18:10:58.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:10:59.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-06-11T18:10:59.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-11T18:11:00.115 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 3 allow_ec_overwrites to true 2024-06-11T18:11:00.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T18:11:00.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:00.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:00.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:00.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:00.137 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:00.138 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:00.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:00.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:00.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:00.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:00.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:00.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:00.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:00.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:00.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:00.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:00.622 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:00.622 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:00.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:00.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:00.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:00.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-06-11T18:11:00.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-06-11T18:11:00.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-06-11T18:11:00.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:00.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:00.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705678 2024-06-11T18:11:00.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705678 2024-06-11T18:11:00.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-38654705678' 2024-06-11T18:11:00.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:00.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:01.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542151 2024-06-11T18:11:01.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542151 2024-06-11T18:11:01.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-38654705678 2-60129542151' 2024-06-11T18:11:01.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:01.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-06-11T18:11:01.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:01.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:01.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-06-11T18:11:01.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:01.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-06-11T18:11:01.074 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836506 2024-06-11T18:11:01.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-11T18:11:01.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:01.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-11T18:11:01.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:01.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705678 2024-06-11T18:11:01.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:01.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:11:01.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705678 2024-06-11T18:11:01.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:01.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705678 2024-06-11T18:11:01.441 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 38654705678 2024-06-11T18:11:01.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2024-06-11T18:11:01.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:11:01.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705678 -lt 38654705678 2024-06-11T18:11:01.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:01.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542151 2024-06-11T18:11:01.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:01.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:11:01.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542151 2024-06-11T18:11:01.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:01.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542151 2024-06-11T18:11:01.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2024-06-11T18:11:01.794 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 60129542151 2024-06-11T18:11:01.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:11:02.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542151 -lt 60129542151 2024-06-11T18:11:02.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:11:02.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:02.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:02.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:11:02.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:11:02.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:11:02.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:11:02.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:11:02.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:11:02.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:11:02.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:11:02.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:11:02.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:11:02.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:02.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:03.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:11:03.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:11:03.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:11:03.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T18:11:03.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-11T18:11:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:03.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:03.305 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:03.305 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:03.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:03.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:03.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:03.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:03.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:03.736 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:03.736 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:03.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:03.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:03.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:03.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-06-11T18:11:03.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-06-11T18:11:03.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-06-11T18:11:03.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:03.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705680 2024-06-11T18:11:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705680 2024-06-11T18:11:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-38654705680' 2024-06-11T18:11:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:04.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542153 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542153 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-38654705680 2-60129542153' 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-06-11T18:11:04.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:04.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:04.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-06-11T18:11:04.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:04.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-06-11T18:11:04.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-11T18:11:04.189 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836507 2024-06-11T18:11:04.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:04.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-11T18:11:04.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:11:05.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:11:05.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:05.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836507 2024-06-11T18:11:05.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:05.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705680 2024-06-11T18:11:05.838 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:05.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:11:05.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705680 2024-06-11T18:11:05.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:05.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705680 2024-06-11T18:11:05.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2024-06-11T18:11:05.842 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 38654705680 2024-06-11T18:11:05.842 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:11:06.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705680 -lt 38654705680 2024-06-11T18:11:06.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:06.180 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542153 2024-06-11T18:11:06.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:06.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:11:06.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542153 2024-06-11T18:11:06.186 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:06.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542153 2024-06-11T18:11:06.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2024-06-11T18:11:06.187 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 60129542153 2024-06-11T18:11:06.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:11:06.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542153 -lt 60129542153 2024-06-11T18:11:06.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:11:06.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:06.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:06.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:11:06.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:11:06.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:11:07.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:11:07.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:11:07.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:07.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:11:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:11:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:11:07.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-11T18:11:07.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:11:07.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:11:07.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:11:08.081 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:11:08.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:11:08.592 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:11:08.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:11:08.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:11:08.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:11:08.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-11T18:11:08.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:11:08.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:11:08.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-11T18:11:08.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-11T18:11:08.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:11:08.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:11:08.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:11:08.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:11:08.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:11:08.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:11:08.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:11:10.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:11:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:11:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:11:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:11:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:11:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:11:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:11:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:11:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:11:11.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:11:11.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:11:11.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:11:11.934 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:11:11.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:11:11.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:11:11.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:11:11.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:11:11.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:11:11.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:11:11.970 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:11.965+0000 7f87435bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:11.972 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:11.969+0000 7f87435bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:11.974 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:11.973+0000 7f87435bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:11:12.279 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:11:12.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:12.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:12.505+0000 7f87435bf7c0 -1 Falling back to public interface 2024-06-11T18:11:12.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:13.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:13.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:13.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:11:13.627 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:13.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:13.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:13.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:14.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:14.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:14.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:11:14.953 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:11:14.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:14.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:15.025 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:15.021+0000 7f87435bf7c0 -1 osd.1 33 log_to_monitors true 2024-06-11T18:11:15.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 34 last_clean_interval [9,33) [v2:127.0.0.1:6810/2429166297,v1:127.0.0.1:6811/2429166297] [v2:127.0.0.1:6812/2429166297,v1:127.0.0.1:6813/2429166297] exists,up 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:16.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:16.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:16.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:16.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:16.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:16.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:16.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:16.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:16.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:16.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:17.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:17.271 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:17.272 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:17.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:17.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:17.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:17.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-06-11T18:11:17.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-06-11T18:11:17.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-06-11T18:11:17.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:17.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:17.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757250 2024-06-11T18:11:17.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757250 2024-06-11T18:11:17.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-163208757250' 2024-06-11T18:11:17.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:17.626 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542157 2024-06-11T18:11:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542157 2024-06-11T18:11:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-163208757250 2-60129542157' 2024-06-11T18:11:17.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:17.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-06-11T18:11:17.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:17.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:17.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:17.775 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-06-11T18:11:17.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-06-11T18:11:17.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-06-11T18:11:17.776 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836511 2024-06-11T18:11:17.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:18.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836511 2024-06-11T18:11:18.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:11:19.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:11:19.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:19.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-06-11T18:11:19.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:19.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-163208757250 2024-06-11T18:11:19.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:19.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:11:19.558 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-163208757250 2024-06-11T18:11:19.558 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:19.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757250 2024-06-11T18:11:19.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-06-11T18:11:19.560 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 163208757250 2024-06-11T18:11:19.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:11:19.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757250 -lt 163208757250 2024-06-11T18:11:19.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:19.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542157 2024-06-11T18:11:19.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:19.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:11:19.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542157 2024-06-11T18:11:19.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:19.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542157 2024-06-11T18:11:19.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2024-06-11T18:11:19.914 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 60129542157 2024-06-11T18:11:19.914 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:11:20.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542157 -lt 60129542157 2024-06-11T18:11:20.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:11:20.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:20.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:20.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:11:20.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:11:20.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:11:20.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:11:20.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:11:20.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:11:20.666 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:11:20.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:11:20.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:11:20.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:11:20.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:20.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:21.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:11:21.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:11:21.806 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:11:21.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:11:22.309 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:11:22.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:11:22.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:11:22.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:11:22.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-11T18:11:22.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:11:22.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:11:22.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-11T18:11:22.500 INFO:tasks.workunit.client.0.smithi012.stderr:1+0 records in 2024-06-11T18:11:22.500 INFO:tasks.workunit.client.0.smithi012.stderr:1+0 records out 2024-06-11T18:11:22.500 INFO:tasks.workunit.client.0.smithi012.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000211451 s, 9.7 MB/s 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:11:22.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:22.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:11:22.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:11:22.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:11:22.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:11:22.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:11:22.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:11:22.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:11:22.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:11:22.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:11:22.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:11:23.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:11:23.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:11:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:11:25.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:11:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:11:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:11:25.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:11:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:11:25.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:11:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:11:25.618 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:11:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:11:25.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:11:25.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:11:25.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:11:25.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:11:25.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:11:25.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:25.649+0000 7fd08b64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:25.657 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:25.653+0000 7fd08b64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:25.658 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:25.657+0000 7fd08b64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:25.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:11:25.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:11:25.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:25.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:26.193 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:26.189+0000 7fd08b64c7c0 -1 Falling back to public interface 2024-06-11T18:11:26.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:27.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:27.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:27.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:11:27.298 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:27.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:27.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:27.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:28.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:28.722 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:28.721+0000 7fd08b64c7c0 -1 osd.0 42 log_to_monitors true 2024-06-11T18:11:28.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:29.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:29.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:29.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:11:29.975 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:11:29.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:29.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:30.599 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 43 last_clean_interval [5,42) [v2:127.0.0.1:6802/3866386264,v1:127.0.0.1:6803/3866386264] [v2:127.0.0.1:6804/3866386264,v1:127.0.0.1:6805/3866386264] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:11:30.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:11:30.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:30.600 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:30.601 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:30.601 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:30.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:30.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:30.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:30.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:30.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:30.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:30.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:30.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:30.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:31.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:31.096 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:31.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:31.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462914 2024-06-11T18:11:31.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462914 2024-06-11T18:11:31.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914' 2024-06-11T18:11:31.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:31.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757254 2024-06-11T18:11:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757254 2024-06-11T18:11:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914 1-163208757254' 2024-06-11T18:11:31.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:31.442 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542160 2024-06-11T18:11:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542160 2024-06-11T18:11:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462914 1-163208757254 2-60129542160' 2024-06-11T18:11:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:31.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-201863462914 2024-06-11T18:11:31.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:31.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:31.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-201863462914 2024-06-11T18:11:31.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:31.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462914 2024-06-11T18:11:31.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2024-06-11T18:11:31.592 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 201863462914 2024-06-11T18:11:31.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 201863462914 2024-06-11T18:11:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:11:32.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:11:32.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:33.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 201863462914 2024-06-11T18:11:33.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:11:34.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:11:34.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:34.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462914 -lt 201863462914 2024-06-11T18:11:34.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:34.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:34.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-163208757254 2024-06-11T18:11:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:11:34.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-163208757254 2024-06-11T18:11:34.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:34.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757254 2024-06-11T18:11:34.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 163208757254' 2024-06-11T18:11:34.590 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 163208757254 2024-06-11T18:11:34.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:11:34.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757254 -lt 163208757254 2024-06-11T18:11:34.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:34.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542160 2024-06-11T18:11:34.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:34.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:11:34.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542160 2024-06-11T18:11:34.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:34.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542160 2024-06-11T18:11:34.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-06-11T18:11:34.923 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 60129542160 2024-06-11T18:11:34.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:11:35.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542161 -lt 60129542160 2024-06-11T18:11:35.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:11:35.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:35.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:11:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:11:35.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:11:35.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:11:35.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:11:35.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:11:35.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:11:35.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:11:36.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:11:36.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:11:36.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:36.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:36.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:11:36.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:11:36.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:11:36.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:11:36.830 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:11:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:11:37.315 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:11:37.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:11:37.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:11:37.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:11:37.638 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-11T18:11:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:11:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:11:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:11:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:11:37.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:11:37.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:11:37.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:11:37.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:11:37.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:11:37.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:11:37.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:11:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:11:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-11T18:11:39.780 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:11:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:11:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:11:40.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:40.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:40.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:11:40.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:11:40.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:11:40.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:11:40.333 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:11:40.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:11:40.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:11:40.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:11:40.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:11:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:11:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:11:40.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:40.365+0000 7f95c461b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:40.370 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:40.369+0000 7f95c461b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:40.372 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:40.369+0000 7f95c461b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:40.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:11:40.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:11:40.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:11:40.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:40.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:40.904 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:40.901+0000 7f95c461b7c0 -1 Falling back to public interface 2024-06-11T18:11:41.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:42.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:43.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:11:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:11:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:43.457 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:43.453+0000 7f95c461b7c0 -1 osd.1 52 log_to_monitors true 2024-06-11T18:11:43.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:44.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:44.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:11:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:11:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [38,52) [v2:127.0.0.1:6810/2086876021,v1:127.0.0.1:6811/2086876021] [v2:127.0.0.1:6812/2086876021,v1:127.0.0.1:6813/2086876021] exists,up 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:45.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:45.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:45.206 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:45.207 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:45.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:45.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:45.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:45.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:45.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:45.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:45.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:45.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:45.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:45.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:45.766 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:45.766 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:45.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:45.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:45.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:45.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462918 2024-06-11T18:11:45.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462918 2024-06-11T18:11:45.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462918' 2024-06-11T18:11:45.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:45.925 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:46.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135874 2024-06-11T18:11:46.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135874 2024-06-11T18:11:46.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462918 1-244813135874' 2024-06-11T18:11:46.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:46.081 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:46.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542164 2024-06-11T18:11:46.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542164 2024-06-11T18:11:46.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-201863462918 1-244813135874 2-60129542164' 2024-06-11T18:11:46.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:46.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-201863462918 2024-06-11T18:11:46.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:46.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:46.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-201863462918 2024-06-11T18:11:46.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:46.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462918 2024-06-11T18:11:46.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 201863462918' 2024-06-11T18:11:46.230 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 201863462918 2024-06-11T18:11:46.231 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:46.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462917 -lt 201863462918 2024-06-11T18:11:46.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:11:47.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:11:47.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:11:47.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462918 -lt 201863462918 2024-06-11T18:11:47.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:47.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135874 2024-06-11T18:11:47.899 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:47.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:11:47.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135874 2024-06-11T18:11:47.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:47.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135874 2024-06-11T18:11:47.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-06-11T18:11:47.903 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135874 2024-06-11T18:11:47.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:11:48.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135874 -lt 244813135874 2024-06-11T18:11:48.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:48.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542164 2024-06-11T18:11:48.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:48.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:11:48.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542164 2024-06-11T18:11:48.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:48.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542164 2024-06-11T18:11:48.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542164' 2024-06-11T18:11:48.231 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 60129542164 2024-06-11T18:11:48.232 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:11:48.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542165 -lt 60129542164 2024-06-11T18:11:48.563 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:11:48.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:48.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:48.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:11:48.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:11:48.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:11:49.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:11:49.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:11:49.333 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:11:49.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-11T18:11:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:11:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:11:50.155 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:11:50.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:11:50.615 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:11:50.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:11:50.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:11:50.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:11:50.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-11T18:11:50.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:11:50.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:11:50.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-11T18:11:51.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-11T18:11:51.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-11T18:11:51.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:11:51.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:11:51.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:11:51.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:11:51.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:11:51.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:11:51.344 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:11:51.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:11:51.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:11:51.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:11:51.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:11:51.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:11:53.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:11:53.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:11:53.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:11:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:11:53.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:11:53.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:11:53.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:11:53.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:11:53.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:11:53.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:11:53.978 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:53.977+0000 7f53f1abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:53.981 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:53.977+0000 7f53f1abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:53.982 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:53.981+0000 7f53f1abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:11:54.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:54.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:11:54.275 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:11:54.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:54.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:54.517 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:54.513+0000 7f53f1abc7c0 -1 Falling back to public interface 2024-06-11T18:11:54.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:55.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:55.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:55.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:11:55.626 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:55.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:55.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:55.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:56.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:11:56.925+0000 7f53f1abc7c0 -1 osd.0 62 log_to_monitors true 2024-06-11T18:11:56.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:56.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:56.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:11:56.963 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:11:56.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:56.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:57.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:11:58.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:11:58.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:11:58.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:11:58.312 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:11:58.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:11:58.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [47,62) [v2:127.0.0.1:6802/1812849002,v1:127.0.0.1:6803/1812849002] [v2:127.0.0.1:6804/1812849002,v1:127.0.0.1:6805/1812849002] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:11:58.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:11:58.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:11:58.759 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:11:58.759 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:11:58.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:11:58.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:11:58.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:11:58.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:11:59.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:11:59.222 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:11:59.223 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:11:59.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:11:59.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:59.223 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:11:59.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808834 2024-06-11T18:11:59.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808834 2024-06-11T18:11:59.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808834' 2024-06-11T18:11:59.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:59.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:11:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135878 2024-06-11T18:11:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135878 2024-06-11T18:11:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808834 1-244813135878' 2024-06-11T18:11:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:11:59.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:11:59.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542168 2024-06-11T18:11:59.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542168 2024-06-11T18:11:59.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808834 1-244813135878 2-60129542168' 2024-06-11T18:11:59.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:11:59.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-287762808834 2024-06-11T18:11:59.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:11:59.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:11:59.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-287762808834 2024-06-11T18:11:59.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:11:59.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808834 2024-06-11T18:11:59.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-06-11T18:11:59.680 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 287762808834 2024-06-11T18:11:59.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:00.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 287762808834 2024-06-11T18:12:00.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:12:01.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:12:01.006 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:01.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808834 -lt 287762808834 2024-06-11T18:12:01.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:01.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135878 2024-06-11T18:12:01.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:01.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:12:01.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135878 2024-06-11T18:12:01.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:01.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135878 2024-06-11T18:12:01.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-06-11T18:12:01.385 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135878 2024-06-11T18:12:01.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:01.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135878 -lt 244813135878 2024-06-11T18:12:01.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:01.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:01.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542168 2024-06-11T18:12:01.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:12:01.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:01.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542168 2024-06-11T18:12:01.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542168 2024-06-11T18:12:01.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542168' 2024-06-11T18:12:01.714 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 60129542168 2024-06-11T18:12:01.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:12:02.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542168 -lt 60129542168 2024-06-11T18:12:02.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:12:02.035 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:02.035 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:02.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:12:02.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:12:02.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:12:02.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:12:02.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:12:02.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:02.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T18:12:03.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:12:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:12:03.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:12:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:12:05.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:12:05.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:12:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:12:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:12:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:12:05.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:12:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:12:05.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:12:05.622 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:12:05.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:12:05.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:12:05.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:12:05.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:12:05.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:12:05.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:12:05.657 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:05.653+0000 7fc8b50057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:05.660 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:05.657+0000 7fc8b50057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:05.662 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:05.661+0000 7fc8b50057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:05.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:06.189+0000 7fc8b50057c0 -1 Falling back to public interface 2024-06-11T18:12:06.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:12:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:07.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:08.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:12:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:12:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:08.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:08.745 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:08.741+0000 7fc8b50057c0 -1 osd.2 69 log_to_monitors true 2024-06-11T18:12:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:09.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:12:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:12:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:09.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:10.484 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [14,69) [v2:127.0.0.1:6818/3683414500,v1:127.0.0.1:6819/3683414500] [v2:127.0.0.1:6820/3683414500,v1:127.0.0.1:6821/3683414500] exists,up 8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:12:10.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:12:10.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:12:10.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:12:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:12:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:12:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:12:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:12:10.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:12:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:12:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:12:10.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:12:10.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:12:10.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:12:10.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:12:10.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:12:10.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:12:10.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:12:10.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:12:10.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:12:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:12:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:12:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:12:10.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:10.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:12:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808837 2024-06-11T18:12:11.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808837 2024-06-11T18:12:11.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808837' 2024-06-11T18:12:11.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:11.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:12:11.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135881 2024-06-11T18:12:11.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135881 2024-06-11T18:12:11.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808837 1-244813135881' 2024-06-11T18:12:11.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:11.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:12:11.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=317827579906 2024-06-11T18:12:11.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 317827579906 2024-06-11T18:12:11.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808837 1-244813135881 2-317827579906' 2024-06-11T18:12:11.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:11.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-287762808837 2024-06-11T18:12:11.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:11.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:12:11.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-287762808837 2024-06-11T18:12:11.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:11.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808837 2024-06-11T18:12:11.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-06-11T18:12:11.413 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 287762808837 2024-06-11T18:12:11.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:11.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808837 -lt 287762808837 2024-06-11T18:12:11.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:11.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135881 2024-06-11T18:12:11.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:11.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:12:11.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135881 2024-06-11T18:12:11.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:11.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135881 2024-06-11T18:12:11.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135881' 2024-06-11T18:12:11.743 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135881 2024-06-11T18:12:11.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:12.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135880 -lt 244813135881 2024-06-11T18:12:12.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:12:13.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:12:13.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:13.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135881 -lt 244813135881 2024-06-11T18:12:13.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:13.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-317827579906 2024-06-11T18:12:13.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:13.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:12:13.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-317827579906 2024-06-11T18:12:13.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:13.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=317827579906 2024-06-11T18:12:13.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-06-11T18:12:13.425 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 317827579906 2024-06-11T18:12:13.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:12:13.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 317827579906 -lt 317827579906 2024-06-11T18:12:13.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:12:13.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:13.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:12:14.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:12:14.173 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:12:14.173 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:12:14.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:12:14.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:12:14.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:14.508 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:14.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:12:14.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:12:14.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:12:14.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:12:14.907 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:12:14.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:12:14.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:12:14.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:12:15.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:12:15.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:12:15.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:12:15.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:12:15.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-11T18:12:15.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:12:15.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:12:15.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-11T18:12:17.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-11T18:12:17.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:12:17.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:12:17.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:12:17.311 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:12:17.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:12:17.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:12:17.313 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:12:17.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:12:17.314 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:12:17.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:12:17.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:12:17.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:12:17.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:12:17.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:17.345+0000 7f170e30e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:17.363 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:17.361+0000 7f170e30e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:17.365 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:17.361+0000 7f170e30e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:12:17.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:17.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:12:17.654 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:12:17.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:17.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:17.901 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:17.897+0000 7f170e30e7c0 -1 Falling back to public interface 2024-06-11T18:12:17.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:18.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:18.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:18.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:12:18.984 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:18.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:18.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:20.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:20.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:20.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:12:20.317 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:12:20.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:20.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:20.460 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:20.457+0000 7f170e30e7c0 -1 osd.2 76 log_to_monitors true 2024-06-11T18:12:20.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:21.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:21.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:21.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:12:21.655 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:12:21.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:21.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:12:22.132 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/1677100178,v1:127.0.0.1:6819/1677100178] [v2:127.0.0.1:6820/1677100178,v1:127.0.0.1:6821/1677100178] exists,up 8477a069-9235-4ade-a413-40a5e44976a8 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:12:22.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:12:22.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:12:22.134 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:12:22.134 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:12:22.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:12:22.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:12:22.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:12:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:12:22.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:12:22.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:12:22.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:12:22.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:12:22.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:12:22.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:12:22.633 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:22.633 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:12:22.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:12:22.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:22.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:12:22.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808841 2024-06-11T18:12:22.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808841 2024-06-11T18:12:22.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808841' 2024-06-11T18:12:22.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:22.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:12:22.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135884 2024-06-11T18:12:22.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135884 2024-06-11T18:12:22.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808841 1-244813135884' 2024-06-11T18:12:22.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:22.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:12:23.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350978 2024-06-11T18:12:23.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350978 2024-06-11T18:12:23.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808841 1-244813135884 2-347892350978' 2024-06-11T18:12:23.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:23.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-287762808841 2024-06-11T18:12:23.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:23.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:12:23.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-287762808841 2024-06-11T18:12:23.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:23.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808841 2024-06-11T18:12:23.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-06-11T18:12:23.094 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 287762808841 2024-06-11T18:12:23.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:23.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808841 -lt 287762808841 2024-06-11T18:12:23.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:23.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135884 2024-06-11T18:12:23.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:23.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:12:23.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135884 2024-06-11T18:12:23.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:23.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135884 2024-06-11T18:12:23.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135884' 2024-06-11T18:12:23.429 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 244813135884 2024-06-11T18:12:23.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:23.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135884 -lt 244813135884 2024-06-11T18:12:23.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:23.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350978 2024-06-11T18:12:23.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:23.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:12:23.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350978 2024-06-11T18:12:23.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:23.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350978 2024-06-11T18:12:23.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-06-11T18:12:23.769 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350978 2024-06-11T18:12:23.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:12:24.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350978 -lt 347892350978 2024-06-11T18:12:24.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:12:24.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:24.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:24.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:12:24.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:12:24.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:12:24.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:12:24.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:12:24.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:12:24.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:12:24.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:12:24.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:12:24.870 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:12:24.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:24.871 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:25.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:12:25.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:12:25.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:12:25.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-11T18:12:25.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:12:25.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-11T18:12:25.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:12:25.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:12:25.681 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:12:25.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:12:26.216 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:12:26.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:12:26.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:12:26.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:12:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-11T18:12:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:12:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:12:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-11T18:12:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2+0 records in 2024-06-11T18:12:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2+0 records out 2024-06-11T18:12:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000273048 s, 15.0 MB/s 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:12:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:12:26.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:12:26.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:12:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:12:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:12:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:12:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:12:29.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:12:29.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:12:29.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:12:29.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:12:29.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:12:29.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:12:29.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:29.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:12:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:12:29.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:12:29.061 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:12:29.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:12:29.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:12:29.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:12:29.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:12:29.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:12:29.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:12:29.096 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:29.093+0000 7f681d0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:29.098 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:29.097+0000 7f681d0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:29.100 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:29.097+0000 7f681d0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:29.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:12:29.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:29.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:29.633 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:29.629+0000 7f681d0467c0 -1 Falling back to public interface 2024-06-11T18:12:29.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:30.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:30.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:30.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:12:30.737 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:30.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:30.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:31.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:32.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:32.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:32.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:12:32.079 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:12:32.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:32.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:32.163 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:32.161+0000 7f681d0467c0 -1 osd.1 85 log_to_monitors true 2024-06-11T18:12:32.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:33.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:33.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:33.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:12:33.464 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:12:33.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:33.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:33.821 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [57,85) [v2:127.0.0.1:6810/3153740992,v1:127.0.0.1:6811/3153740992] [v2:127.0.0.1:6812/3153740992,v1:127.0.0.1:6813/3153740992] exists,up 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:12:33.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:12:33.823 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:12:33.823 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:12:33.823 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:12:33.823 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:12:33.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:12:33.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:12:33.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:12:33.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:12:33.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:12:33.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:12:33.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:12:33.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:12:34.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:12:34.283 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:34.283 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:12:34.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:12:34.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:34.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:12:34.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=287762808844 2024-06-11T18:12:34.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 287762808844 2024-06-11T18:12:34.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808844' 2024-06-11T18:12:34.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:34.439 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:12:34.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056642 2024-06-11T18:12:34.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056642 2024-06-11T18:12:34.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808844 1-386547056642' 2024-06-11T18:12:34.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:34.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:12:34.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350981 2024-06-11T18:12:34.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350981 2024-06-11T18:12:34.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-287762808844 1-386547056642 2-347892350981' 2024-06-11T18:12:34.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:34.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-287762808844 2024-06-11T18:12:34.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:34.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:12:34.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-287762808844 2024-06-11T18:12:34.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:34.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=287762808844 2024-06-11T18:12:34.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-06-11T18:12:34.733 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 287762808844 2024-06-11T18:12:34.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:35.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808843 -lt 287762808844 2024-06-11T18:12:35.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:12:36.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:12:36.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:36.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 287762808844 -lt 287762808844 2024-06-11T18:12:36.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:36.383 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-386547056642 2024-06-11T18:12:36.384 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:36.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:12:36.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-386547056642 2024-06-11T18:12:36.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:36.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056642 2024-06-11T18:12:36.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-06-11T18:12:36.389 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 386547056642 2024-06-11T18:12:36.389 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:36.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 386547056642 -lt 386547056642 2024-06-11T18:12:36.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:36.708 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350981 2024-06-11T18:12:36.708 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:36.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:12:36.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350981 2024-06-11T18:12:36.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:36.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350981 2024-06-11T18:12:36.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350981' 2024-06-11T18:12:36.712 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350981 2024-06-11T18:12:36.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:12:37.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350981 -lt 347892350981 2024-06-11T18:12:37.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:12:37.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:37.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:37.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:12:37.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:12:37.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:12:37.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:12:37.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:12:37.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:37.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:38.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:12:38.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:12:38.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:12:38.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:12:38.577 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:12:38.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:12:39.080 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:12:39.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:12:39.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:12:39.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:12:39.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-11T18:12:39.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-11T18:12:39.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:12:39.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:12:39.240 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:12:39.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:12:39.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:12:39.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:12:39.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:12:40.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:12:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-11T18:12:42.469 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:12:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:12:42.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:12:42.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:12:42.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:12:42.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:12:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:12:42.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:12:42.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:12:42.474 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:12:42.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:12:42.476 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:12:42.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:12:42.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:12:42.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:12:42.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:12:42.510 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:42.509+0000 7fb6b128b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:42.513 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:42.509+0000 7fb6b128b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:42.515 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:42.513+0000 7fb6b128b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:42.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:12:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:43.057+0000 7fb6b128b7c0 -1 Falling back to public interface 2024-06-11T18:12:43.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:44.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:44.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:44.170 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:12:44.170 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:44.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:44.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:12:44.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:45.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:45.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:45.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:12:45.507 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:12:45.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:45.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:12:45.776 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:45.773+0000 7fb6b128b7c0 -1 osd.0 94 log_to_monitors true 2024-06-11T18:12:45.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:12:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:12:46.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:46.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:12:47.337 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 99 up_thru 99 down_at 95 last_clean_interval [67,94) [v2:127.0.0.1:6802/757503286,v1:127.0.0.1:6803/757503286] [v2:127.0.0.1:6804/757503286,v1:127.0.0.1:6805/757503286] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:12:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:12:47.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:12:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:12:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:12:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:12:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:12:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:12:47.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:12:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:12:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:12:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:12:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:47.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:12:48.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=425201762306 2024-06-11T18:12:48.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 425201762306 2024-06-11T18:12:48.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762306' 2024-06-11T18:12:48.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:48.003 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:12:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056646 2024-06-11T18:12:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056646 2024-06-11T18:12:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762306 1-386547056646' 2024-06-11T18:12:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:48.149 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:12:48.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350985 2024-06-11T18:12:48.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350985 2024-06-11T18:12:48.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762306 1-386547056646 2-347892350985' 2024-06-11T18:12:48.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:48.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-425201762306 2024-06-11T18:12:48.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:48.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:12:48.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-425201762306 2024-06-11T18:12:48.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:48.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=425201762306 2024-06-11T18:12:48.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2024-06-11T18:12:48.289 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 425201762306 2024-06-11T18:12:48.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:48.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 425201762306 2024-06-11T18:12:48.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:12:49.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:12:49.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:12:49.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762306 -lt 425201762306 2024-06-11T18:12:49.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:49.951 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-386547056646 2024-06-11T18:12:49.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:49.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:12:49.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-386547056646 2024-06-11T18:12:49.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:49.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056646 2024-06-11T18:12:49.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 386547056646' 2024-06-11T18:12:49.956 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 386547056646 2024-06-11T18:12:49.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:12:50.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 386547056646 -lt 386547056646 2024-06-11T18:12:50.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:12:50.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:12:50.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350985 2024-06-11T18:12:50.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:12:50.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350985 2024-06-11T18:12:50.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:12:50.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350985 2024-06-11T18:12:50.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-06-11T18:12:50.303 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350985 2024-06-11T18:12:50.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:12:50.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350985 -lt 347892350985 2024-06-11T18:12:50.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:12:50.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:50.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:51.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:12:51.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:12:51.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:12:51.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:12:51.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:12:51.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:12:51.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:12:51.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:12:51.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:12:51.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:12:51.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:12:51.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:12:51.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:12:51.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:12:51.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:12:52.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-11T18:12:54.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:12:54.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:12:54.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:12:54.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:12:54.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:12:54.356 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:54.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:12:54.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:12:54.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:12:54.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:12:54.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:12:54.358 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:12:54.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:12:54.359 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:12:54.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:12:54.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:12:54.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:12:54.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:12:54.393 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:54.389+0000 7f9f694607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:54.396 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:54.393+0000 7f9f694607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:54.398 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:54.397+0000 7f9f694607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:54.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:54.936 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:54.933+0000 7f9f694607c0 -1 Falling back to public interface 2024-06-11T18:12:55.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:56.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:56.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:56.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:12:56.037 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:56.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:56.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:56.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:57.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:57.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:57.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:12:57.372 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:12:57.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:57.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:12:57.609+0000 7f9f694607c0 -1 osd.1 101 log_to_monitors true 2024-06-11T18:12:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:12:58.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:12:58.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:12:58.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:12:58.744 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:12:58.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:12:58.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [90,101) [v2:127.0.0.1:6810/4213633853,v1:127.0.0.1:6811/4213633853] [v2:127.0.0.1:6812/4213633853,v1:127.0.0.1:6813/4213633853] exists,up 11c09347-73fb-4120-91f6-2d2a666028cb 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:12:59.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:12:59.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:12:59.092 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:12:59.092 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:12:59.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:12:59.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:12:59.093 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:12:59.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:12:59.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:12:59.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:12:59.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:12:59.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:12:59.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:12:59.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:12:59.563 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:12:59.564 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:12:59.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:12:59.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:59.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:12:59.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=425201762309 2024-06-11T18:12:59.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 425201762309 2024-06-11T18:12:59.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762309' 2024-06-11T18:12:59.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:59.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:12:59.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=455266533378 2024-06-11T18:12:59.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 455266533378 2024-06-11T18:12:59.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762309 1-455266533378' 2024-06-11T18:12:59.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:12:59.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:13:00.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350988 2024-06-11T18:13:00.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350988 2024-06-11T18:13:00.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-425201762309 1-455266533378 2-347892350988' 2024-06-11T18:13:00.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:00.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:00.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-425201762309 2024-06-11T18:13:00.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:13:00.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-425201762309 2024-06-11T18:13:00.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:00.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=425201762309 2024-06-11T18:13:00.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 425201762309' 2024-06-11T18:13:00.073 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 425201762309 2024-06-11T18:13:00.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:13:00.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762308 -lt 425201762309 2024-06-11T18:13:00.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:13:01.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:13:01.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:13:01.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762310 -lt 425201762309 2024-06-11T18:13:01.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:01.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-455266533378 2024-06-11T18:13:01.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:01.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:13:01.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-455266533378 2024-06-11T18:13:01.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:01.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=455266533378 2024-06-11T18:13:01.738 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-06-11T18:13:01.738 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 455266533378 2024-06-11T18:13:01.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:13:02.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 455266533378 -lt 455266533378 2024-06-11T18:13:02.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:02.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350988 2024-06-11T18:13:02.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:02.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:13:02.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350988 2024-06-11T18:13:02.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:02.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350988 2024-06-11T18:13:02.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-06-11T18:13:02.080 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350988 2024-06-11T18:13:02.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:13:02.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350989 -lt 347892350988 2024-06-11T18:13:02.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:13:02.420 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:02.420 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:02.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:13:02.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:13:02.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:13:03.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:13:03.182 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:13:03.182 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:03.183 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:03.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:13:03.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:13:03.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:13:03.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-11T18:13:03.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:13:03.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:13:03.996 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:13:04.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:13:04.439 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:13:04.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:13:04.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:13:04.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:13:04.679 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-11T18:13:04.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-11T18:13:04.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-11T18:13:04.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-11T18:13:04.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-11T18:13:04.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-11T18:13:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:13:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:13:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:13:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:13:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:13:04.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:13:04.874 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:13:04.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:13:04.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:13:04.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:13:05.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-11T18:13:07.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:13:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:13:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:13:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:13:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:13:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:13:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:13:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:13:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:13:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:13:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:13:07.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:13:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:13:07.334 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:13:07.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:13:07.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:13:07.335 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:13:07.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:13:07.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:13:07.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:13:07.362 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:07.356+0000 7f26cf6987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:07.372 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:07.368+0000 7f26cf6987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:07.368+0000 7f26cf6987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:13:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:13:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:07.916 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:07.912+0000 7f26cf6987c0 -1 Falling back to public interface 2024-06-11T18:13:07.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:08.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:08.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:08.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:13:08.995 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:13:08.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:08.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:09.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:10.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:10.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:10.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:13:10.370 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:13:10.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:10.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:10.484+0000 7f26cf6987c0 -1 osd.0 110 log_to_monitors true 2024-06-11T18:13:10.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:11.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:13:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:13:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [99,110) [v2:127.0.0.1:6802/561011228,v1:127.0.0.1:6803/561011228] [v2:127.0.0.1:6804/561011228,v1:127.0.0.1:6805/561011228] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:13:12.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:13:12.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:13:12.085 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:13:12.085 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:13:12.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:13:12.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:13:12.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:13:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:13:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:13:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:13:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:13:12.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:13:12.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:13:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:13:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:13:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:13:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:13:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:12.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:13:12.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=493921239042 2024-06-11T18:13:12.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 493921239042 2024-06-11T18:13:12.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-493921239042' 2024-06-11T18:13:12.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:12.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:13:12.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=455266533382 2024-06-11T18:13:12.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 455266533382 2024-06-11T18:13:12.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-493921239042 1-455266533382' 2024-06-11T18:13:12.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:12.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:13:13.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350992 2024-06-11T18:13:13.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350992 2024-06-11T18:13:13.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-493921239042 1-455266533382 2-347892350992' 2024-06-11T18:13:13.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:13.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-493921239042 2024-06-11T18:13:13.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:13.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:13:13.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-493921239042 2024-06-11T18:13:13.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:13.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=493921239042 2024-06-11T18:13:13.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-06-11T18:13:13.024 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 493921239042 2024-06-11T18:13:13.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:13:13.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 493921239042 -lt 493921239042 2024-06-11T18:13:13.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:13.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-455266533382 2024-06-11T18:13:13.361 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:13.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:13:13.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-455266533382 2024-06-11T18:13:13.364 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:13.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=455266533382 2024-06-11T18:13:13.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 455266533382' 2024-06-11T18:13:13.366 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 455266533382 2024-06-11T18:13:13.366 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:13:13.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 455266533382 -lt 455266533382 2024-06-11T18:13:13.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:13.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350992 2024-06-11T18:13:13.698 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:13.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:13:13.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350992 2024-06-11T18:13:13.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:13.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350992 2024-06-11T18:13:13.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350992' 2024-06-11T18:13:13.702 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350992 2024-06-11T18:13:13.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:13:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350992 -lt 347892350992 2024-06-11T18:13:14.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:13:14.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:14.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:14.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:13:14.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:13:14.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:13:14.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:13:14.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:13:14.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:13:14.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:13:14.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:13:14.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:13:14.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:13:14.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:14.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:15.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:13:15.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:13:15.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:13:15.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:13:15.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:13:15.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:13:15.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:13:15.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:13:15.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:13:15.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:13:15.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:13:15.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:13:15.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:13:15.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:13:15.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:13:15.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:13:15.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:13:15.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:13:15.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:13:15.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:13:15.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:13:15.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:13:15.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:13:15.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:13:15.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:13:15.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:13:15.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-11T18:13:17.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:13:17.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:13:17.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:13:17.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:13:17.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:13:17.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:13:17.973 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:13:17.973 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:17.973 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:17.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:17.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:13:17.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:13:17.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:13:17.975 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:13:17.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:13:17.975 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:13:17.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:13:17.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:13:17.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:13:17.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:13:18.011 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:18.008+0000 7f55836867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:18.014 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:18.012+0000 7f55836867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:18.015 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:18.012+0000 7f55836867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:18.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:13:18.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:18.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:18.549 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:18.544+0000 7f55836867c0 -1 Falling back to public interface 2024-06-11T18:13:18.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:19.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:19.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:19.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:13:19.643 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:13:19.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:19.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:19.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:20.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:20.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:20.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:13:20.970 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:13:20.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:20.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:21.100 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:21.096+0000 7f55836867c0 -1 osd.0 117 log_to_monitors true 2024-06-11T18:13:21.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:22.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:22.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:22.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:13:22.313 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:13:22.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:22.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:22.778 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/3074464275,v1:127.0.0.1:6803/3074464275] [v2:127.0.0.1:6804/3074464275,v1:127.0.0.1:6805/3074464275] exists,up 8a9c778f-5dbc-4282-bc11-64a28c12f14f 2024-06-11T18:13:22.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:13:22.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:13:22.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:13:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:13:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:13:22.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:13:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:13:22.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:13:22.898 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:13:23.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:13:23.300 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:13:23.300 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:13:23.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:13:23.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:23.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:13:23.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=523986010114 2024-06-11T18:13:23.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 523986010114 2024-06-11T18:13:23.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-523986010114' 2024-06-11T18:13:23.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:23.458 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:13:23.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=455266533385 2024-06-11T18:13:23.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 455266533385 2024-06-11T18:13:23.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-523986010114 1-455266533385' 2024-06-11T18:13:23.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:13:23.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=347892350995 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 347892350995 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-523986010114 1-455266533385 2-347892350995' 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-523986010114 2024-06-11T18:13:23.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:23.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:13:23.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-523986010114 2024-06-11T18:13:23.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:23.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=523986010114 2024-06-11T18:13:23.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-06-11T18:13:23.752 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 523986010114 2024-06-11T18:13:23.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:13:24.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 523986010114 2024-06-11T18:13:24.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:13:25.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:13:25.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:13:25.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 523986010114 -lt 523986010114 2024-06-11T18:13:25.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:25.420 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-455266533385 2024-06-11T18:13:25.420 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:25.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:13:25.423 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-455266533385 2024-06-11T18:13:25.424 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:25.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=455266533385 2024-06-11T18:13:25.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-06-11T18:13:25.425 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 455266533385 2024-06-11T18:13:25.426 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:13:25.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 455266533385 -lt 455266533385 2024-06-11T18:13:25.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:13:25.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-347892350995 2024-06-11T18:13:25.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:13:25.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:13:25.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-347892350995 2024-06-11T18:13:25.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:13:25.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=347892350995 2024-06-11T18:13:25.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 347892350995' 2024-06-11T18:13:25.762 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.2 seq 347892350995 2024-06-11T18:13:25.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:13:26.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 347892350995 -lt 347892350995 2024-06-11T18:13:26.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:13:26.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:26.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:26.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:13:26.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:13:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:13:26.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:13:26.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:13:26.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:13:26.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:13:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:13:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:13:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:13:27.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-11T18:13:27.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-11T18:13:27.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:13:27.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-11T18:13:27.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:13:27.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-11T18:13:27.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-11T18:13:27.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-11T18:13:27.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=3.0 2024-06-11T18:13:27.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 3.0 2024-06-11T18:13:27.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:27.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:27.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:27.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:27.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:27.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 3.0 2024-06-11T18:13:28.218 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 3.0 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:13:28.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:13:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:28.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:28.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:28.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:29.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:29.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:29.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:13:29.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:29.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:29.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:29.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:29.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:29.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:30.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:13:30.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:30.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:30.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:30.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:31.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:31.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:32.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:32.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:32.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:13:32.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:32.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:32.226 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:32.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:32.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:32.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:33.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:13:33.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:33.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:13:33.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:33.567 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:13:33.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:13:28.613143+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:33.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:13:33.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-11T18:13:33.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:13:33.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-11T18:13:33.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:13:33.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-11T18:13:33.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-11T18:13:34.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:13:34.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:13:34.051 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:13:34.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:13:34.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:13:34.095 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:13:34.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:13:34.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:13:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:13:34.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: local pgid=3.0 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:34.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:34.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:34.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:34.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: local last_scrub=2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:34.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-11T18:13:34.755 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-11T18:13:34.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: pg_deep_scrub: wait_for_scrub 3.0 2024-06-11T18:10:58.400374+0000 last_deep_scrub_stamp 2024-06-11T18:13:34.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:13:34.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:34.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-11T18:13:34.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:13:34.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:34.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:34.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:34.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:34.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:34.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:35.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:35.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:36.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:36.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:36.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:37.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:37.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:37.443 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:37.443 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:37.443 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:37.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:37.444 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:37.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:37.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:38.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:38.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:38.777 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:38.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:39.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:10:58.400374+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:39.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:13:40.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:13:40.111 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:13:40.112 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:13:40.112 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:13:40.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:13:40.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:13:40.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:13:40.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:13:35.871457+0000 '>' 2024-06-11T18:10:58.400374+0000 2024-06-11T18:13:40.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:13:40.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-11T18:13:40.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:13:40.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-11T18:13:40.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:13:40.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-11T18:13:40.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-11T18:13:40.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:13:40.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-06-11T18:13:40.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:13:40.606 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:13:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:13:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:13:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:13:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:13:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:13:40.648 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-11T18:13:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:13:40.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:13:40.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:13:40.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-11T18:13:40.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-11T18:13:40.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-11T18:13:41.136 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' removed 2024-06-11T18:13:41.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:13:41.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:13:41.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:13:41.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:13:41.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:13:41.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:13:41.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:13:41.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:13:41.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:13:41.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:13:41.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:13:41.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:13:41.475 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:13:41.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:13:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:13:41.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:13:41.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:13:41.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:13:41.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:13:41.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:13:41.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:13:41.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:13:41.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:13:41.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:13:41.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:41.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:41.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:13:41.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:13:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:13:41.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:13:41.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:13:41.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:13:41.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:13:41.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:13:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:13:41.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:13:41.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:13:41.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:13:41.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:13:41.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:13:41.529 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:13:41.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:13:41.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:13:41.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:13:41.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:13:41.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:13:41.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:13:41.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:13:41.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:13:41.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:41.536 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:41.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:13:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:13:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:13:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:13:41.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:13:41.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:41.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:41.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:13:41.541 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:13:41.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:13:41.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:13:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:13:41.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:13:41.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-11T18:13:42.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:13:42.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:13:42.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:13:42.064 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:13:42.065 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:42.065 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:42.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:42.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-11T18:13:42.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:13:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:13:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:13:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:13:42.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:13:42.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:13:42.328 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:13:42.328 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:13:42.328 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:13:42.328 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:13:42.329 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:42.329 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:42.329 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:13:42.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:13:42.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:13:42.445 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:13:42.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:13:42.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:13:42.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:42.447 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:13:42.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:13:42.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:13:42.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-11T18:13:42.552 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:13:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:13:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:13:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:13:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:13:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:13:44.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:13:44.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:13:44.253 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:13:44.254 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:13:44.254 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:44.254 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:44.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:44.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:13:44.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:13:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:13:44.298 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:13:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:13:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:13:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:13:44.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:13:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:13:44.300 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:13:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8afc0318-49ee-4091-8db9-38fea222c77d' 2024-06-11T18:13:44.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:13:44.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYk2hmhQrvEhAAZBv9tBMBag2t3D6E1osrYQ== 2024-06-11T18:13:44.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYk2hmhQrvEhAAZBv9tBMBag2t3D6E1osrYQ=="}' 2024-06-11T18:13:44.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8afc0318-49ee-4091-8db9-38fea222c77d -i td/osd-scrub-repair/0/new.json 2024-06-11T18:13:44.619 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:13:44.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:13:44.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYk2hmhQrvEhAAZBv9tBMBag2t3D6E1osrYQ== --osd-uuid 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:13:44.675 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.672+0000 7f36652107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.700+0000 7f36652107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.700+0000 7f36652107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:44.706 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.704+0000 7f36652107c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:13:44.706 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.704+0000 7f36652107c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:13:44.706 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.704+0000 7f36652107c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:13:44.805 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:44.800+0000 7f36652107c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:13:52.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:13:52.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:13:52.602 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:13:52.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:13:52.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:13:53.010 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:13:53.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:13:53.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:13:53.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:13:53.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:13:53.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:13:53.045 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:53.040+0000 7fd9232a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:53.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:53.080+0000 7fd9232a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:53.086 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:53.084+0000 7fd9232a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:13:53.316 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:13:53.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:13:53.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:13:53.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:13:53.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:13:53.317 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:13:53.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:53.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:13:53.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:53.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:53.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:53.660 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:53.656+0000 7fd9232a07c0 -1 Falling back to public interface 2024-06-11T18:13:54.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:54.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:54.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:13:54.646 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:13:54.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:54.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:54.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:55.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:55.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:55.974 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:13:55.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:13:55.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:55.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:56.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:56.557 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:13:56.552+0000 7fd9232a07c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:13:57.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:57.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:57.302 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:13:57.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:13:57.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:57.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:57.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:13:58.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:13:58.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:13:58.721 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:13:58.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:13:58.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:13:58.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:13:59.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:00.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:00.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:00.051 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:14:00.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:14:00.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:00.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:00.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:01.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:01.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:01.403 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:14:01.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:14:01.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:01.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:01.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:02.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:02.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:02.905 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:14:02.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:14:02.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:02.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:03.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:14:04.241 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:14:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:04.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:05.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:05.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:05.679 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:14:05.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:14:05.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:05.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:06.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:07.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:07.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:07.017 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:14:07.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:14:07.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:07.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:08.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:08.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:08.443 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:14:08.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:14:08.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:08.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:08.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:09.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:09.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:09.942 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:14:09.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:14:09.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:09.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:10.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:11.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:11.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:11.280 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:14:11.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:14:11.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:11.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:11.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:12.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:12.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:12.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:14:12.616 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:14:12.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:12.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:12.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:13.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:13.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:13.947 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:14:13.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:14:13.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:13.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:14.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:15.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:15.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:15.285 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:14:15.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:14:15.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:15.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:15.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:16.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:16.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:16.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:14:16.915 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:14:16.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:16.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:17.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:18.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:18.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:18.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:14:18.695 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:14:18.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:18.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:19.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:20.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:20.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:20.295 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:14:20.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:14:20.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:20.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:21.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:22.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:22.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:22.005 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:14:22.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:14:22.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:22.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:22.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:23.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:14:23.681 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:14:23.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:23.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:25.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:25.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:25.098 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:14:25.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:14:25.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:25.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:25.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:26.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:26.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:26.759 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:14:26.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:14:26.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:26.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:27.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:28.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:28.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:28.268 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:14:28.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:14:28.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:28.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:14:28.604 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2755608587,v1:127.0.0.1:6803/2755608587] [v2:127.0.0.1:6804/2755608587,v1:127.0.0.1:6805/2755608587] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:14:28.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:14:28.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:14:28.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:14:28.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:14:28.608 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:14:28.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:14:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:14:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:14:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:14:28.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:14:28.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:14:28.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:14:28.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:14:28.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b2541df2-1a8b-47b0-9ae2-f9de633bebc8' 2024-06-11T18:14:28.611 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:14:28.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:14:28.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAElGhm32bNJRAAGQqw+UVbM0b0EWg3z22EwQ== 2024-06-11T18:14:28.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAElGhm32bNJRAAGQqw+UVbM0b0EWg3z22EwQ=="}' 2024-06-11T18:14:28.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2541df2-1a8b-47b0-9ae2-f9de633bebc8 -i td/osd-scrub-repair/1/new.json 2024-06-11T18:14:29.182 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:14:29.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:14:29.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAElGhm32bNJRAAGQqw+UVbM0b0EWg3z22EwQ== --osd-uuid b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:14:29.246 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.240+0000 7f39af6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:29.326 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.324+0000 7f39af6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:29.328 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.324+0000 7f39af6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:29.329 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.324+0000 7f39af6a07c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:14:29.329 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.324+0000 7f39af6a07c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:14:29.329 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.324+0000 7f39af6a07c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:14:29.414 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:29.412+0000 7f39af6a07c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:14:37.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:14:37.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:14:37.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:14:37.611 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:14:37.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:14:38.026 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:14:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:14:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:14:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:14:38.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:14:38.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:14:38.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:38.056+0000 7f4ee7cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:38.106 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:38.104+0000 7f4ee7cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:38.108 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:38.104+0000 7f4ee7cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:14:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:14:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:14:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:14:38.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:38.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:38.669 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:38.664+0000 7f4ee7cae7c0 -1 Falling back to public interface 2024-06-11T18:14:38.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:39.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:39.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:39.716 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:14:39.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:14:39.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:39.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:40.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:41.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:41.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:41.049 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:14:41.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:14:41.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:41.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:41.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:41.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:14:41.648+0000 7f4ee7cae7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:14:42.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:42.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:42.388 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:14:42.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:14:42.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:42.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:42.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:43.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:43.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:43.897 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:14:43.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:14:43.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:43.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:44.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:45.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:45.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:45.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:14:45.354 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:14:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:45.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:46.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:46.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:46.821 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:14:46.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:14:46.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:46.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:47.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:48.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:48.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:48.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:14:48.160 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:14:48.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:48.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:48.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:49.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:49.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:49.500 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:14:49.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:14:49.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:49.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:49.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:50.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:50.885 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:50.885 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:14:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:14:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:51.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:52.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:52.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:52.213 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:14:52.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:14:52.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:52.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:52.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:53.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:53.574 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:14:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:14:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:53.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:53.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:54.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:54.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:54.909 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:14:54.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:14:54.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:54.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:55.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:56.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:56.248 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:56.249 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:14:56.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:14:56.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:56.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:56.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:57.587 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:14:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:14:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:58.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:14:59.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:14:59.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:14:59.013 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:14:59.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:14:59.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:14:59.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:14:59.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:00.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:00.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:00.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:15:00.355 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:15:00.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:00.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:00.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:01.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:01.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:01.823 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:15:01.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:15:01.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:01.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:02.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:03.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:03.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:03.443 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:15:03.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:15:03.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:03.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:04.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:05.075 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:15:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:15:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:05.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:06.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:15:06.487 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:15:06.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:06.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:06.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:07.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:07.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:07.921 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:15:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:15:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:07.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:08.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:09.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:09.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:09.736 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:15:09.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:15:09.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:09.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:11.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:11.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:11.472 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:15:11.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:15:11.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:11.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:13.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:13.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:13.533 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:15:13.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:15:13.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:13.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:14.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:15:15.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:15:15.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:15:15.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-11T18:15:15.150 INFO:tasks.workunit.client.0.smithi012.stdout:25 2024-06-11T18:15:15.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:15:15.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:15:15.545 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1941768756,v1:127.0.0.1:6811/1941768756] [v2:127.0.0.1:6812/1941768756,v1:127.0.0.1:6813/1941768756] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:15:15.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:15:15.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:15:15.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:15:15.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-06-11T18:15:15.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:15:15.862 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:15:15.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:15:15.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:15:16.369 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:15:16.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:15:17.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:15:20.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-11T18:15:20.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:15:20.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:15:20.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:15:20.484 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:15:20.485 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:15:20.485 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:15:20.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:15:20.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:15:20.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:15:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:15:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:15:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:15:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:15:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:15:20.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:15:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:15:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:15:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:15:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:15:20.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:15:21.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-06-11T18:15:21.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-06-11T18:15:21.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-06-11T18:15:21.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:15:21.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:15:21.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672967 2024-06-11T18:15:21.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672967 2024-06-11T18:15:21.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672967' 2024-06-11T18:15:21.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:15:21.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-06-11T18:15:21.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:15:21.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:15:21.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-06-11T18:15:21.324 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:15:21.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-06-11T18:15:21.325 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836499 2024-06-11T18:15:21.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-11T18:15:21.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:15:21.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-11T18:15:21.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:15:21.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672967 2024-06-11T18:15:21.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:15:21.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:15:21.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672967 2024-06-11T18:15:21.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:15:21.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672967 2024-06-11T18:15:21.649 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672967 2024-06-11T18:15:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-11T18:15:21.650 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:15:21.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672966 -lt 42949672967 2024-06-11T18:15:21.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:15:22.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:15:22.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:15:23.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672966 -lt 42949672967 2024-06-11T18:15:23.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:15:24.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:15:24.306 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:15:24.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-11T18:15:24.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:15:24.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:15:24.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:15:25.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:15:25.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:15:25.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:15:25.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:15:25.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:15:25.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:15:25.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:15:25.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:15:25.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:15:25.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:15:25.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:15:25.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:15:25.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:15:25.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:15:25.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:15:25.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-06-11T18:15:25.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-11T18:15:26.318 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'foo' created 2024-06-11T18:15:26.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:15:27.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-06-11T18:15:27.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-11T18:15:27.882 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'csr_pool' created 2024-06-11T18:15:27.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:15:28.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-11T18:15:28.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:15:28.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:15:28.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:15:28.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:15:28.904 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:15:28.904 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:15:28.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:15:28.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:15:28.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:15:29.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:15:29.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:15:29.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:15:29.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:15:29.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:15:29.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:15:29.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:15:29.472 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:15:29.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:15:29.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:15:29.472 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:15:29.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-06-11T18:15:29.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-06-11T18:15:29.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-06-11T18:15:29.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:15:29.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:15:29.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-06-11T18:15:29.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-06-11T18:15:29.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672969' 2024-06-11T18:15:29.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:15:29.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-06-11T18:15:29.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:15:29.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:15:29.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-06-11T18:15:29.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:15:29.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-06-11T18:15:29.853 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836502 2024-06-11T18:15:29.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-11T18:15:29.854 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:15:30.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836502 2024-06-11T18:15:30.239 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:15:31.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:15:31.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:15:31.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-06-11T18:15:31.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:15:31.631 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-06-11T18:15:31.631 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:15:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:15:31.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-06-11T18:15:31.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:15:31.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-06-11T18:15:31.636 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672969 2024-06-11T18:15:31.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-11T18:15:31.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:15:32.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672970 -lt 42949672969 2024-06-11T18:15:32.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:15:32.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:15:32.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:15:32.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:15:32.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:15:32.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:15:32.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:15:32.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:15:32.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:15:32.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:15:32.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:15:32.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:15:32.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:15:32.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:15:32.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:15:33.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:15:33.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:15:33.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:15:33.305 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-11T18:15:33.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:33.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-11T18:15:33.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-11T18:15:33.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-11T18:15:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:33.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:33.878 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:33.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:34.397 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:34.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:34.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:34.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:34.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-11T18:15:34.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:35.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:35.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:35.567 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:35.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:36.069 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:36.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:36.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:36.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:36.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-11T18:15:36.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-06-11T18:15:36.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:36.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:36.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:37.314 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:37.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:37.832 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:38.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-06-11T18:15:38.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:38.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:39.002 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:39.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:39.562 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:39.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:39.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:39.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:39.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-06-11T18:15:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-06-11T18:15:40.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:40.774 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:40.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:41.301 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:41.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-06-11T18:15:41.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-06-11T18:15:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:42.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:42.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:42.545 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:42.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:43.148 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:43.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:43.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:43.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:43.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-06-11T18:15:43.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:43.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:44.393 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:44.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:44.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:45.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-06-11T18:15:45.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-06-11T18:15:45.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:46.165 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:46.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:46.667 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-06-11T18:15:47.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-06-11T18:15:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:47.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:47.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:47.946 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:47.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:48.481 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:48.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:48.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:48.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:48.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-06-11T18:15:49.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-06-11T18:15:49.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:49.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-11T18:15:49.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-06-11T18:15:49.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:49.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:49.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-06-11T18:15:49.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:49.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:49.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:49.751 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:49.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:50.328 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:50.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:50.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:50.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:50.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-06-11T18:15:50.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-06-11T18:15:51.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:51.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:51.566 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:51.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:52.084 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:52.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:52.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:52.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:52.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-06-11T18:15:52.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-06-11T18:15:52.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:52.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:53.279 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:53.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:53.864 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:53.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:53.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:53.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:54.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-06-11T18:15:54.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-06-11T18:15:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-11T18:15:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-06-11T18:15:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:54.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:54.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-06-11T18:15:54.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:54.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:54.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:55.134 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:55.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:55.661 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:55.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:55.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:55.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:56.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-06-11T18:15:56.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-06-11T18:15:56.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:56.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:57.283 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:57.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:57.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:57.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:57.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:15:58.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-06-11T18:15:58.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-06-11T18:15:58.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:15:58.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:15:59.096 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:15:59.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:15:59.639 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:15:59.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:15:59.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:15:59.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:16:00.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-06-11T18:16:00.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-06-11T18:16:00.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:00.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-11T18:16:00.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:16:00.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:16:00.817 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:16:00.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:16:01.319 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:16:01.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:16:01.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:16:01.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:16:01.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-06-11T18:16:01.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-06-11T18:16:02.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:02.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-11T18:16:02.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-06-11T18:16:02.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:16:02.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:16:02.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-06-11T18:16:02.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:16:02.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:16:02.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:16:02.614 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:16:02.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:16:03.115 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:16:03.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:16:03.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:16:03.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:16:03.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-06-11T18:16:03.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-06-11T18:16:03.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:16:03.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:16:04.394 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:16:04.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:16:05.038 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:16:05.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:16:05.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:16:05.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:16:05.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-06-11T18:16:05.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:16:05.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:16:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:16:06.308 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:16:06.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:16:06.818 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:16:06.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:16:06.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:16:06.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:16:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-06-11T18:16:07.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-06-11T18:16:07.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-06-11T18:16:07.740 INFO:tasks.workunit.client.0.smithi012.stderr:1025+0 records in 2024-06-11T18:16:07.740 INFO:tasks.workunit.client.0.smithi012.stderr:1025+0 records out 2024-06-11T18:16:07.740 INFO:tasks.workunit.client.0.smithi012.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00420626 s, 250 MB/s 2024-06-11T18:16:07.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-06-11T18:16:08.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-06-11T18:16:08.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-11T18:16:08.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-11T18:16:08.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-11T18:16:08.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-11T18:16:08.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:16:08.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-11T18:16:08.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-06-11T18:16:08.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-11T18:16:08.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-11T18:16:08.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-11T18:16:08.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:16:08.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-06-11T18:16:08.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-06-11T18:16:08.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:16:08.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:16:08.760 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:16:08.761 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:08.761 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:08.761 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:16:08.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-11T18:16:08.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61904/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-11T18:16:08.867 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:16:08.867 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-11T18:16:08.867 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:16:08.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-06-11T18:16:08.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:16:08.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:16:08.881 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:16:08.881 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:08.882 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:08.882 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:16:08.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-11T18:16:08.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-11T18:16:08.995 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:16:08.995 INFO:tasks.workunit.client.0.smithi012.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-11T18:16:08.995 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:16:09.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-11T18:16:09.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: local pgid=3.0 2024-06-11T18:16:09.010 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:16:09.010 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:16:09.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:16:09.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:16:09.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:16:09.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: local last_scrub=2024-06-11T18:15:27.718608+0000 2024-06-11T18:16:09.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-11T18:16:09.618 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: pg_deep_scrub: wait_for_scrub 3.0 2024-06-11T18:15:27.718608+0000 last_deep_scrub_stamp 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:15:27.718608+0000 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:16:09.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:16:09.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:16:09.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:16:09.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:16:09.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:16:09.637 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:16:09.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:15:27.718608+0000 '>' 2024-06-11T18:15:27.718608+0000 2024-06-11T18:16:09.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:16:10.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:16:10.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:16:10.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:16:10.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:16:10.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:16:10.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:16:10.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:16:11.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:15:27.718608+0000 2024-06-11T18:16:11.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:16:11.267 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-11T18:16:11.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:11.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-11T18:16:11.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:16:11.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:16:11.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:16:11.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:16:11.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:16:11.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:13.659 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-06-11T18:16:14.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:16:14.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:16:14.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:16:14.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:16:14.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:16:14.242 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:16:14.243 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:16:14.243 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:14.243 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:14.243 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:16:14.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:16:14.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:16:14.245 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:16:14.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:16:14.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:16:14.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:16:14.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:16:14.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:16:14.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:16:14.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:14.275+0000 7f4ceaaa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:14.283 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:14.279+0000 7f4ceaaa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:14.284 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:14.279+0000 7f4ceaaa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:16:14.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:14.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:16:14.584 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:16:14.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:14.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:14.820 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:14.815+0000 7f4ceaaa37c0 -1 Falling back to public interface 2024-06-11T18:16:14.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:15.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:16:15.939 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:16:15.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:15.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:16.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:16:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:16:17.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:17.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:17.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:17.902 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:17.899+0000 7f4ceaaa37c0 -1 osd.1 64 log_to_monitors true 2024-06-11T18:16:18.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:18.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:18.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:16:18.715 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:16:18.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:18.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/3480203125,v1:127.0.0.1:6811/3480203125] [v2:127.0.0.1:6812/3480203125,v1:127.0.0.1:6813/3480203125] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:16:19.132 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:16:19.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:16:19.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:16:19.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:16:19.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:16:19.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:16:19.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:16:19.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:16:19.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:16:19.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:16:19.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:16:19.648 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:16:19.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:16:19.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:19.649 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:16:19.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-06-11T18:16:19.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-06-11T18:16:19.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-06-11T18:16:19.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:19.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:16:20.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=292057776130 2024-06-11T18:16:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 292057776130 2024-06-11T18:16:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-292057776130' 2024-06-11T18:16:20.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:20.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-06-11T18:16:20.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:20.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:16:20.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-06-11T18:16:20.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:20.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-06-11T18:16:20.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-11T18:16:20.057 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836513 2024-06-11T18:16:20.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:20.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836513 2024-06-11T18:16:20.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:16:21.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:16:21.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:21.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-11T18:16:21.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:21.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-292057776130 2024-06-11T18:16:21.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:21.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:16:21.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:21.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-292057776130 2024-06-11T18:16:21.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=292057776130 2024-06-11T18:16:21.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-06-11T18:16:21.803 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 292057776130 2024-06-11T18:16:21.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:16:22.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 292057776130 -lt 292057776130 2024-06-11T18:16:22.200 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:16:22.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:22.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:22.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:16:22.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:22.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:23.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:16:23.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:23.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:23.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:16:23.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:16:23.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:16:23.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:23.534 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:23.534 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:23.534 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:23.534 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:23.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:23.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:23.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:16:23.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:23.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:23.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:16:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:16:24.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:16:24.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:16:24.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:16:24.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:16:24.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:16:24.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:16:24.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:16:24.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:16:24.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:16:24.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:24.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:24.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:24.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:25.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:16:25.228 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:25.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:25.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:16:25.619 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:16:25.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:16:25.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:16:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:16:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:16:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T18:16:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:16:26.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T18:16:26.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:16:26.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:26.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:26.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:26.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:26.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:26.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:26.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:26.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:16:26.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:26.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:26.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:27.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:16:27.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:16:27.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:16:27.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:27.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-11T18:16:27.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:16:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:16:27.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:16:27.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:16:27.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:16:27.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:16:27.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:16:27.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:16:28.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:16:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:16:30.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:16:30.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:30.958 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:30.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:16:30.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:16:30.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:16:30.961 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:16:30.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:16:30.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:16:30.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:16:30.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:16:30.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:16:30.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:16:31.003 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:30.999+0000 7f381cc897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:31.006 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:31.003+0000 7f381cc897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:31.008 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:31.003+0000 7f381cc897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:16:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:16:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:16:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:31.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:31.540 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:31.535+0000 7f381cc897c0 -1 Falling back to public interface 2024-06-11T18:16:31.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:32.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:16:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:16:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:32.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:32.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:33.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:33.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:33.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:16:33.986 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:16:33.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:33.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:34.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:34.484 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:34.479+0000 7f381cc897c0 -1 osd.0 70 log_to_monitors true 2024-06-11T18:16:35.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:35.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:35.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:16:35.320 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:16:35.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:35.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/1214997242,v1:127.0.0.1:6803/1214997242] [v2:127.0.0.1:6804/1214997242,v1:127.0.0.1:6805/1214997242] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:16:35.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:16:35.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:16:35.669 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:16:35.669 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:16:35.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:16:35.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:16:35.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:16:35.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:16:35.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:16:35.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:16:35.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:16:35.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:16:35.787 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:16:36.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:16:36.254 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:16:36.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:16:36.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:36.254 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:16:36.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=317827579906 2024-06-11T18:16:36.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 317827579906 2024-06-11T18:16:36.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-317827579906' 2024-06-11T18:16:36.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:36.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:16:36.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=292057776134 2024-06-11T18:16:36.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 292057776134 2024-06-11T18:16:36.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-317827579906 1-292057776134' 2024-06-11T18:16:36.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:36.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-317827579906 2024-06-11T18:16:36.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:36.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:16:36.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-317827579906 2024-06-11T18:16:36.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:36.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=317827579906 2024-06-11T18:16:36.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-06-11T18:16:36.617 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 317827579906 2024-06-11T18:16:36.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:36.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 317827579906 2024-06-11T18:16:36.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:16:37.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:16:37.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:38.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 317827579906 -lt 317827579906 2024-06-11T18:16:38.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:38.270 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-292057776134 2024-06-11T18:16:38.270 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:38.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:16:38.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-292057776134 2024-06-11T18:16:38.272 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:38.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=292057776134 2024-06-11T18:16:38.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 292057776134' 2024-06-11T18:16:38.274 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 292057776134 2024-06-11T18:16:38.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:16:38.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 292057776134 -lt 292057776134 2024-06-11T18:16:38.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:16:38.645 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:38.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:39.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:16:39.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:39.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:39.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:39.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:39.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:39.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:39.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:39.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:16:39.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:39.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:39.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:16:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:16:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:16:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-11T18:16:39.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:16:39.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:16:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:16:39.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:16:39.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:16:39.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:16:39.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:16:39.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:16:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:16:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-06-11T18:16:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:16:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:16:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:16:41.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:16:41.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:16:41.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-06-11T18:16:42.570 INFO:tasks.workunit.client.0.smithi012.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-06-11T18:16:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:16:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:16:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:16:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:16:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:16:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:16:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:16:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:16:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:16:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:16:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:16:43.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:16:43.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:16:43.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:16:43.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:16:43.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:16:43.146 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:43.139+0000 7fe06d8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:43.148 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:43.143+0000 7fe06d8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:43.150 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:43.143+0000 7fe06d8157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:43.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:16:43.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:16:43.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:43.697 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:43.691+0000 7fe06d8157c0 -1 Falling back to public interface 2024-06-11T18:16:43.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:46.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:46.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:46.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:16:46.150 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:16:46.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:46.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:46.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:46.471+0000 7fe06d8157c0 -1 osd.1 75 log_to_monitors true 2024-06-11T18:16:46.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:47.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:47.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:47.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:16:47.491 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:16:47.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:47.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:16:47.962 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/2563340598,v1:127.0.0.1:6811/2563340598] [v2:127.0.0.1:6812/2563340598,v1:127.0.0.1:6813/2563340598] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:16:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:16:47.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:16:48.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:16:48.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:16:48.512 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:16:48.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:16:48.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:48.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:16:48.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=317827579909 2024-06-11T18:16:48.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 317827579909 2024-06-11T18:16:48.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-317827579909' 2024-06-11T18:16:48.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:16:48.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:16:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=339302416386 2024-06-11T18:16:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 339302416386 2024-06-11T18:16:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-06-11T18:16:48.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:48.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-317827579909 2024-06-11T18:16:48.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:48.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:16:48.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-317827579909 2024-06-11T18:16:48.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:48.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=317827579909 2024-06-11T18:16:48.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-06-11T18:16:48.852 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 317827579909 2024-06-11T18:16:48.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:49.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 317827579908 -lt 317827579909 2024-06-11T18:16:49.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:16:50.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:16:50.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:16:50.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 317827579909 -lt 317827579909 2024-06-11T18:16:50.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:16:50.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-339302416386 2024-06-11T18:16:50.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:16:50.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:16:50.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-339302416386 2024-06-11T18:16:50.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:16:50.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=339302416386 2024-06-11T18:16:50.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-06-11T18:16:50.490 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 339302416386 2024-06-11T18:16:50.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:16:50.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 339302416386 -lt 339302416386 2024-06-11T18:16:50.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:16:50.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:50.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:51.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:16:51.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:16:51.295 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:16:51.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:16:51.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:16:51.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:16:51.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-11T18:16:52.004 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:52.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:16:52.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:16:53.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-11T18:16:55.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:16:55.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:16:55.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:16:55.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:16:55.281 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:16:55.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:16:55.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:16:55.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:16:55.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:16:55.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:16:55.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:16:55.317 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:55.311+0000 7fb9749147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:55.320 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:55.315+0000 7fb9749147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:55.322 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:55.319+0000 7fb9749147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:16:55.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:16:55.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:16:55.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:55.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:55.869 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:55.863+0000 7fb9749147c0 -1 Falling back to public interface 2024-06-11T18:16:56.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:57.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:57.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:57.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:16:57.026 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:16:57.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:57.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:57.361 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:58.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:58.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:58.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:16:58.363 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:16:58.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:58.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:16:58.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:16:58.724 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:16:58.719+0000 7fb9749147c0 -1 osd.0 80 log_to_monitors true 2024-06-11T18:16:59.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:16:59.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:16:59.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:16:59.706 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:16:59.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:16:59.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:00.269 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/1795450468,v1:127.0.0.1:6803/1795450468] [v2:127.0.0.1:6804/1795450468,v1:127.0.0.1:6805/1795450468] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:17:00.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:17:00.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:17:00.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:17:00.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:17:00.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:17:00.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:17:00.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:17:00.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:17:00.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:17:00.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:17:00.728 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:17:00.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:17:00.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:00.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:17:00.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252866 2024-06-11T18:17:00.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252866 2024-06-11T18:17:00.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-360777252866' 2024-06-11T18:17:00.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:00.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:17:01.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=339302416389 2024-06-11T18:17:01.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 339302416389 2024-06-11T18:17:01.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-06-11T18:17:01.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:01.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-360777252866 2024-06-11T18:17:01.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:01.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:17:01.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-360777252866 2024-06-11T18:17:01.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252866 2024-06-11T18:17:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-06-11T18:17:01.049 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 360777252866 2024-06-11T18:17:01.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:01.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 360777252866 2024-06-11T18:17:01.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:02.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:17:02.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:02.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252866 -lt 360777252866 2024-06-11T18:17:02.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:02.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-339302416389 2024-06-11T18:17:02.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:02.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:17:02.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-339302416389 2024-06-11T18:17:02.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:02.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=339302416389 2024-06-11T18:17:02.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-06-11T18:17:02.711 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 339302416389 2024-06-11T18:17:02.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:17:03.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 339302416389 -lt 339302416389 2024-06-11T18:17:03.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:17:03.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:03.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:17:03.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:03.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:03.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:17:03.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:03.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:03.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:17:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:17:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:17:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:17:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-11T18:17:04.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-06-11T18:17:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:17:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:17:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-11T18:17:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:17:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:17:04.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:17:05.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:17:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:17:07.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:17:07.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:17:07.664 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:17:07.664 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:17:07.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:17:07.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:17:07.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:17:07.666 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:17:07.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:17:07.667 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:17:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:17:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:17:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:17:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:17:07.702 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:07.695+0000 7f887fbe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:07.705 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:07.699+0000 7f887fbe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:07.706 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:07.703+0000 7f887fbe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:07.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:17:07.997 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:07.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:08.241 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:08.235+0000 7f887fbe27c0 -1 Falling back to public interface 2024-06-11T18:17:08.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:09.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:09.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:09.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:17:09.369 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:17:09.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:09.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:09.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:10.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:10.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:10.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:17:10.698 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:17:10.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:10.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:11.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:11.074 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:11.067+0000 7f887fbe27c0 -1 osd.1 85 log_to_monitors true 2024-06-11T18:17:12.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:12.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:12.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:17:12.036 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:17:12.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:12.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [79,85) [v2:127.0.0.1:6810/1424266226,v1:127.0.0.1:6811/1424266226] [v2:127.0.0.1:6812/1424266226,v1:127.0.0.1:6813/1424266226] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:17:12.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:17:12.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:17:12.593 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:17:12.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:17:12.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:17:12.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:17:12.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:17:12.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:17:12.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:17:12.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:17:12.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:17:12.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:17:12.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:17:13.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:17:13.187 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:17:13.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:17:13.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:13.187 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:17:13.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=360777252869 2024-06-11T18:17:13.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 360777252869 2024-06-11T18:17:13.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-360777252869' 2024-06-11T18:17:13.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:13.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:17:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089346 2024-06-11T18:17:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089346 2024-06-11T18:17:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-360777252869 1-382252089346' 2024-06-11T18:17:13.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:13.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-360777252869 2024-06-11T18:17:13.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:13.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:17:13.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-360777252869 2024-06-11T18:17:13.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:13.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=360777252869 2024-06-11T18:17:13.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-06-11T18:17:13.494 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 360777252869 2024-06-11T18:17:13.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:13.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252868 -lt 360777252869 2024-06-11T18:17:13.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:14.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:17:14.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:15.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252868 -lt 360777252869 2024-06-11T18:17:15.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:16.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:17:16.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:16.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 360777252870 -lt 360777252869 2024-06-11T18:17:16.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:16.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-382252089346 2024-06-11T18:17:16.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:16.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:17:16.505 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-382252089346 2024-06-11T18:17:16.505 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:16.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089346 2024-06-11T18:17:16.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-06-11T18:17:16.506 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 382252089346 2024-06-11T18:17:16.510 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:17:16.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089346 -lt 382252089346 2024-06-11T18:17:16.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:17:16.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:16.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:17.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:17:17.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:17.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:17.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:17.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:17.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:17.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:17.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:17.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:17:17.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:17.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:17.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:18.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:17:18.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:17:18.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:17:18.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:17:18.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-11T18:17:18.123 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-06-11T18:17:18.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:17:18.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:17:18.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:17:18.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:17:18.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:17:18.127 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:17:18.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:17:18.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:17:18.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:17:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:17:19.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-11T18:17:21.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:17:21.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:17:21.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:17:21.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:17:21.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:17:21.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:17:21.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:17:21.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:17:21.459 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:17:21.460 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:17:21.460 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:17:21.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:17:21.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:17:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:17:21.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:17:21.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:17:21.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:17:21.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:17:21.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:17:21.498 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:21.490+0000 7ff06902a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:21.510 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:21.502+0000 7ff06902a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:21.511 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:21.506+0000 7ff06902a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:21.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:22.044 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:22.038+0000 7ff06902a7c0 -1 Falling back to public interface 2024-06-11T18:17:22.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:23.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:23.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:23.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:17:23.171 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:17:23.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:23.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:23.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:24.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:24.847 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:24.842+0000 7ff06902a7c0 -1 osd.0 90 log_to_monitors true 2024-06-11T18:17:25.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:26.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:26.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:26.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:17:26.038 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:17:26.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:26.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 94 up_thru 94 down_at 91 last_clean_interval [84,90) [v2:127.0.0.1:6802/3917267275,v1:127.0.0.1:6803/3917267275] [v2:127.0.0.1:6804/3917267275,v1:127.0.0.1:6805/3917267275] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:17:26.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:17:26.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:17:26.596 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:17:26.596 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:17:26.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:17:26.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:17:26.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:17:26.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:17:26.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:17:26.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:17:26.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:17:26.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:17:26.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:17:27.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:17:27.152 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:17:27.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:17:27.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:27.152 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:17:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925826 2024-06-11T18:17:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925826 2024-06-11T18:17:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-403726925826' 2024-06-11T18:17:27.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:27.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:17:27.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089350 2024-06-11T18:17:27.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089350 2024-06-11T18:17:27.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-403726925826 1-382252089350' 2024-06-11T18:17:27.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:27.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-403726925826 2024-06-11T18:17:27.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:27.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:17:27.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-403726925826 2024-06-11T18:17:27.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:27.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925826 2024-06-11T18:17:27.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 403726925826' 2024-06-11T18:17:27.545 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 403726925826 2024-06-11T18:17:27.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:27.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 403726925826 2024-06-11T18:17:27.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:28.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:17:28.863 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:29.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 403726925826 2024-06-11T18:17:29.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:30.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:17:30.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:30.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925826 -lt 403726925826 2024-06-11T18:17:30.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:30.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:30.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-382252089350 2024-06-11T18:17:30.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:17:30.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-382252089350 2024-06-11T18:17:30.539 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:30.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089350 2024-06-11T18:17:30.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-06-11T18:17:30.541 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 382252089350 2024-06-11T18:17:30.541 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:17:30.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089350 -lt 382252089350 2024-06-11T18:17:30.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:17:30.910 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:30.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:31.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:17:31.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:31.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:31.303 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:31.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:31.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:31.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:31.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:31.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:17:31.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:31.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:31.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:32.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:17:32.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:17:32.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:17:32.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-06-11T18:17:32.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:17:32.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-11T18:17:32.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-06-11T18:17:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:17:32.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:17:32.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:17:32.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:17:32.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:17:32.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:17:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:17:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-11T18:17:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:17:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:17:33.340 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:17:33.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:17:33.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:17:33.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:17:35.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:17:35.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:17:35.048 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:17:35.048 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:17:35.048 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:17:35.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:35.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:17:35.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:17:35.050 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:17:35.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:17:35.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:17:35.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:17:35.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:17:35.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:17:35.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:17:35.085 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:35.078+0000 7f0a9a3bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:35.088 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:35.082+0000 7f0a9a3bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:35.089 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:35.082+0000 7f0a9a3bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:35.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:35.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:35.633 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:35.626+0000 7f0a9a3bf7c0 -1 Falling back to public interface 2024-06-11T18:17:35.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:37.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:38.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:38.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:38.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:17:38.059 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:17:38.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:38.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:38.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:38.397 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:38.394+0000 7f0a9a3bf7c0 -1 osd.1 95 log_to_monitors true 2024-06-11T18:17:39.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:39.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:39.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:17:39.392 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:17:39.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:39.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [89,95) [v2:127.0.0.1:6810/3780799446,v1:127.0.0.1:6811/3780799446] [v2:127.0.0.1:6812/3780799446,v1:127.0.0.1:6813/3780799446] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:17:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:17:39.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:17:39.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:17:39.980 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:17:39.980 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:17:39.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:17:39.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:17:39.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:17:40.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:17:40.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:17:40.446 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:17:40.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:17:40.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:40.446 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:17:40.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925830 2024-06-11T18:17:40.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925830 2024-06-11T18:17:40.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-403726925830' 2024-06-11T18:17:40.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:40.597 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:17:40.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=425201762306 2024-06-11T18:17:40.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 425201762306 2024-06-11T18:17:40.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-403726925830 1-425201762306' 2024-06-11T18:17:40.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:40.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-403726925830 2024-06-11T18:17:40.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:40.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:17:40.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-403726925830 2024-06-11T18:17:40.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:40.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925830 2024-06-11T18:17:40.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 403726925830' 2024-06-11T18:17:40.762 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 403726925830 2024-06-11T18:17:40.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925828 -lt 403726925830 2024-06-11T18:17:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:42.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:17:42.083 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:42.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925830 -lt 403726925830 2024-06-11T18:17:42.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:42.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-425201762306 2024-06-11T18:17:42.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:42.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:17:42.417 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-425201762306 2024-06-11T18:17:42.417 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:42.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=425201762306 2024-06-11T18:17:42.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-06-11T18:17:42.419 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 425201762306 2024-06-11T18:17:42.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:17:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762306 -lt 425201762306 2024-06-11T18:17:42.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:17:42.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:42.747 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:43.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:17:43.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:43.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:43.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:43.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:43.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:43.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:43.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:17:43.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:43.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:43.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:17:43.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:17:43.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:17:43.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:43.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:43.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:43.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:43.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:43.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:43.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:17:44.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:44.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:44.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-11T18:17:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T18:17:44.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:17:44.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:17:44.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:17:44.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:17:44.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:17:44.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:17:44.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:17:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:17:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:17:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:17:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:17:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:17:45.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:17:45.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:45.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:45.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:45.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:45.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:45.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:45.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:45.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:17:45.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:45.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:45.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:46.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:17:46.048 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:17:46.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:17:46.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:17:46.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T18:17:46.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:17:46.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T18:17:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:17:46.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:46.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:46.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:46.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:46.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:46.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:46.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:17:47.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:47.185 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:47.186 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:47.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-11T18:17:47.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-11T18:17:47.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:17:47.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:17:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:17:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:17:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:17:47.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:17:47.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:17:47.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:17:47.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:17:47.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-06-11T18:17:47.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:17:47.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-06-11T18:17:48.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:17:48.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:17:48.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:17:48.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:17:48.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:17:48.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:17:48.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:17:48.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:17:49.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:17:49.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:17:49.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:17:49.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:17:49.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-11T18:17:49.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-06-11T18:17:49.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:17:49.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:17:49.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-06-11T18:17:49.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-06-11T18:17:49.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-11T18:17:49.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-11T18:17:49.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:17:49.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:17:49.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:17:49.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:17:49.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:17:49.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:17:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:17:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:17:49.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:17:50.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:17:52.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:17:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:17:52.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:17:52.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:17:52.453 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:17:52.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:17:52.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:17:52.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:17:52.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:17:52.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:17:52.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:17:52.489 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:52.486+0000 7f513ca687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:52.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:52.490+0000 7f513ca687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:52.494 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:52.490+0000 7f513ca687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:17:52.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:17:52.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:52.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:53.032 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:53.030+0000 7f513ca687c0 -1 Falling back to public interface 2024-06-11T18:17:53.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:54.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:54.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:54.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:17:54.167 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:17:54.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:54.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:54.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:55.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:55.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:55.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:17:55.493 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:17:55.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:55.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:55.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:17:55.879 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:17:55.878+0000 7f513ca687c0 -1 osd.0 101 log_to_monitors true 2024-06-11T18:17:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:17:56.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:17:56.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:17:56.875 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:17:56.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:17:56.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:17:57.215 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [94,101) [v2:127.0.0.1:6802/4119064321,v1:127.0.0.1:6803/4119064321] [v2:127.0.0.1:6804/4119064321,v1:127.0.0.1:6805/4119064321] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:17:57.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:17:57.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:17:57.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:17:57.217 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:17:57.217 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:17:57.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:17:57.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:17:57.217 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:17:57.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:17:57.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:17:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:17:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:17:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:17:57.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:17:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:17:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:17:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:17:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:57.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:17:58.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566082 2024-06-11T18:17:58.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566082 2024-06-11T18:17:58.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566082' 2024-06-11T18:17:58.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:17:58.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=425201762310 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 425201762310 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566082 1-425201762310' 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-450971566082 2024-06-11T18:17:58.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:58.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:17:58.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-450971566082 2024-06-11T18:17:58.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:58.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566082 2024-06-11T18:17:58.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-06-11T18:17:58.171 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 450971566082 2024-06-11T18:17:58.171 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:58.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 450971566082 2024-06-11T18:17:58.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:17:59.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:17:59.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:17:59.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566082 -lt 450971566082 2024-06-11T18:17:59.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:17:59.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:17:59.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-425201762310 2024-06-11T18:17:59.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:17:59.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-425201762310 2024-06-11T18:17:59.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:17:59.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=425201762310 2024-06-11T18:17:59.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 425201762310' 2024-06-11T18:17:59.829 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 425201762310 2024-06-11T18:17:59.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:18:00.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762311 -lt 425201762310 2024-06-11T18:18:00.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:18:00.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:00.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:00.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:18:00.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:00.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:01.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:18:01.004 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:01.005 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:01.005 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:01.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:18:01.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:18:01.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:18:01.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:18:01.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:18:01.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:18:01.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:18:01.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:18:01.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:18:01.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:01.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-11T18:18:04.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:18:04.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:18:04.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:18:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:18:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:18:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:18:04.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:18:04.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:18:04.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:18:04.224 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:18:04.224 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:18:04.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:18:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:18:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:18:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:18:04.226 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:18:04.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:18:04.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:18:04.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:18:04.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:18:04.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:18:04.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:18:04.262 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:04.258+0000 7f52507797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:04.264 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:04.262+0000 7f52507797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:04.266 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:04.266+0000 7f52507797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:18:04.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:18:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:18:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:04.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:04.809 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:04.806+0000 7f52507797c0 -1 Falling back to public interface 2024-06-11T18:18:04.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:18:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:18:05.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:05.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:06.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:18:07.211 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:18:07.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:07.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:07.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:07.716 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:07.714+0000 7f52507797c0 -1 osd.1 106 log_to_monitors true 2024-06-11T18:18:08.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:08.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:08.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:18:08.587 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:18:08.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:08.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:09.016 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/842551773,v1:127.0.0.1:6811/842551773] [v2:127.0.0.1:6812/842551773,v1:127.0.0.1:6813/842551773] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:18:09.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:18:09.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:18:09.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:18:09.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:18:09.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:18:09.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:18:09.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:18:09.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:18:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:18:09.018 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:18:09.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:18:09.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:18:09.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:18:09.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:18:09.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:18:09.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:18:09.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:18:09.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:18:09.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:18:09.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:18:09.623 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:18:09.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:18:09.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:09.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:18:09.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566085 2024-06-11T18:18:09.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566085 2024-06-11T18:18:09.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566085' 2024-06-11T18:18:09.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:09.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:18:10.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=472446402562 2024-06-11T18:18:10.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 472446402562 2024-06-11T18:18:10.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-06-11T18:18:10.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:10.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-450971566085 2024-06-11T18:18:10.052 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:10.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:18:10.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-450971566085 2024-06-11T18:18:10.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:10.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566085 2024-06-11T18:18:10.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-06-11T18:18:10.056 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 450971566085 2024-06-11T18:18:10.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:10.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566084 -lt 450971566085 2024-06-11T18:18:10.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:18:11.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:18:11.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:11.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566085 -lt 450971566085 2024-06-11T18:18:11.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:11.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-472446402562 2024-06-11T18:18:11.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:11.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:18:11.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-472446402562 2024-06-11T18:18:11.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:11.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=472446402562 2024-06-11T18:18:11.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-06-11T18:18:11.733 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 472446402562 2024-06-11T18:18:11.734 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:18:12.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 472446402562 -lt 472446402562 2024-06-11T18:18:12.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:18:12.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:12.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:12.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:18:12.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:12.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:12.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:12.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:12.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:12.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:12.549 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:12.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:18:12.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:12.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:12.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:18:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:18:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:18:13.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:18:13.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:18:13.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:18:13.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:18:13.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:13.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:13.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:13.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:13.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:13.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:13.409 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:13.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:18:13.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:13.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:13.745 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:14.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:18:14.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:18:14.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:18:14.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:18:14.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:18:14.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:18:14.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:18:15.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:15.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-11T18:18:17.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:18:17.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:18:17.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:18:17.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:18:17.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:18:17.807 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:18:17.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:18:17.808 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:18:17.808 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:18:17.808 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:18:17.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:18:17.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:18:17.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:18:17.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:18:17.810 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:18:17.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:18:17.811 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:18:17.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:18:17.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:18:17.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:18:17.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:18:17.847 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:17.846+0000 7fdc74a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:17.849 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:17.846+0000 7fdc74a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:17.851 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:17.850+0000 7fdc74a9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:18.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:18.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:18.382+0000 7fdc74a9a7c0 -1 Falling back to public interface 2024-06-11T18:18:18.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:19.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:19.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:20.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:20.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:20.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:18:20.846 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:18:20.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:20.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:21.180 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:21.178+0000 7fdc74a9a7c0 -1 osd.1 112 log_to_monitors true 2024-06-11T18:18:21.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:22.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:22.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:22.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:18:22.183 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:18:22.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:22.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 116 up_thru 110 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1068566170,v1:127.0.0.1:6811/1068566170] [v2:127.0.0.1:6812/1068566170,v1:127.0.0.1:6813/1068566170] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:18:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:18:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:18:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:18:22.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:18:22.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:18:22.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:18:22.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:18:22.726 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:18:22.727 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:18:22.727 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:18:22.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:18:22.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:18:22.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:18:22.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:18:22.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:18:22.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:18:22.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:18:22.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:18:22.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:18:23.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:18:23.386 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:18:23.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:18:23.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:23.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:18:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566089 2024-06-11T18:18:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566089 2024-06-11T18:18:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566089' 2024-06-11T18:18:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:23.681 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:18:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=498216206338 2024-06-11T18:18:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 498216206338 2024-06-11T18:18:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566089 1-498216206338' 2024-06-11T18:18:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:23.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-450971566089 2024-06-11T18:18:23.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:23.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:18:23.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-450971566089 2024-06-11T18:18:23.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:23.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566089 2024-06-11T18:18:23.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 450971566089' 2024-06-11T18:18:23.904 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 450971566089 2024-06-11T18:18:23.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566088 -lt 450971566089 2024-06-11T18:18:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:18:25.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:18:25.302 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:25.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566089 -lt 450971566089 2024-06-11T18:18:25.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:25.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-498216206338 2024-06-11T18:18:25.659 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:25.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:18:25.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-498216206338 2024-06-11T18:18:25.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:25.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=498216206338 2024-06-11T18:18:25.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-06-11T18:18:25.664 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 498216206338 2024-06-11T18:18:25.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:18:26.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 498216206338 -lt 498216206338 2024-06-11T18:18:26.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:18:26.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:26.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:18:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:26.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:26.462 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:26.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T18:18:26.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:26.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:26.822 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:18:27.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:18:27.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:18:27.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:27.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:27.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:27.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:27.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:27.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:27.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:27.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:18:27.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:27.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:27.639 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:18:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:18:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:18:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-11T18:18:28.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:28.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-11T18:18:28.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:18:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:18:28.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:18:28.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:18:28.041 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:18:28.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:18:28.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:18:28.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:29.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-06-11T18:18:31.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:18:31.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:18:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:18:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:18:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:18:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:18:31.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:18:31.268 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:18:31.269 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:18:31.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:18:31.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:18:31.270 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:18:31.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:18:31.271 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:18:31.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:18:31.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:18:31.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:18:31.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:18:31.308 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:31.306+0000 7f2033a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:31.354 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:31.354+0000 7f2033a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:31.356 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:31.354+0000 7f2033a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:31.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:31.896 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:31.894+0000 7f2033a957c0 -1 Falling back to public interface 2024-06-11T18:18:31.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:32.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:32.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:32.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:18:32.931 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:18:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:32.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:33.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:34.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:34.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:34.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:18:34.270 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:18:34.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:34.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:34.489 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:34.486+0000 7f2033a957c0 -1 osd.1 118 log_to_monitors true 2024-06-11T18:18:34.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:35.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:35.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:35.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:18:35.663 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:18:35.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:35.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:36.137 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 122 up_thru 122 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6810/2887180324,v1:127.0.0.1:6811/2887180324] [v2:127.0.0.1:6812/2887180324,v1:127.0.0.1:6813/2887180324] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:18:36.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:18:36.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:18:36.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:18:36.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:18:36.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:18:36.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:18:36.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:18:36.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:18:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:18:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:18:36.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:18:36.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:18:36.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:18:36.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:18:36.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:18:36.716 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:18:36.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:18:36.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:36.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:18:36.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566093 2024-06-11T18:18:36.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566093 2024-06-11T18:18:36.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566093' 2024-06-11T18:18:36.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:36.873 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:18:37.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=523986010114 2024-06-11T18:18:37.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 523986010114 2024-06-11T18:18:37.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566093 1-523986010114' 2024-06-11T18:18:37.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:37.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-450971566093 2024-06-11T18:18:37.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:37.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:18:37.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-450971566093 2024-06-11T18:18:37.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:37.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566093 2024-06-11T18:18:37.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 450971566093' 2024-06-11T18:18:37.090 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 450971566093 2024-06-11T18:18:37.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:37.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566091 -lt 450971566093 2024-06-11T18:18:37.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:18:38.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:18:38.414 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:38.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566093 -lt 450971566093 2024-06-11T18:18:38.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:38.749 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-523986010114 2024-06-11T18:18:38.749 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:38.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:18:38.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-523986010114 2024-06-11T18:18:38.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:38.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=523986010114 2024-06-11T18:18:38.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 523986010114' 2024-06-11T18:18:38.754 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 523986010114 2024-06-11T18:18:38.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:18:39.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 523986010114 -lt 523986010114 2024-06-11T18:18:39.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:18:39.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:39.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:39.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:18:39.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:39.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:39.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:39.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:39.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:39.544 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:39.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:39.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:18:39.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:39.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:39.856 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:18:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:18:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:18:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-06-11T18:18:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-11T18:18:40.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:18:40.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:18:40.387 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:18:40.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:18:40.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:18:40.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:18:40.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:18:40.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:18:43.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:18:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:18:43.499 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:18:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:18:43.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:18:43.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:18:43.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:18:43.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:18:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:43.534+0000 7f87f14947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:43.538+0000 7f87f14947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:43.539 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:43.538+0000 7f87f14947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:43.832 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:44.087 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:44.086+0000 7f87f14947c0 -1 Falling back to public interface 2024-06-11T18:18:44.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:45.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:45.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:45.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:18:45.233 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:18:45.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:45.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:45.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:18:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:18:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:46.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:46.897 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:46.894+0000 7f87f14947c0 -1 osd.1 124 log_to_monitors true 2024-06-11T18:18:46.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:47.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:47.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:47.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:18:47.906 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:18:47.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:47.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:18:48.306 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 128 up_thru 122 down_at 125 last_clean_interval [122,124) [v2:127.0.0.1:6810/3544703991,v1:127.0.0.1:6811/3544703991] [v2:127.0.0.1:6812/3544703991,v1:127.0.0.1:6813/3544703991] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:18:48.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:18:48.308 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:18:48.308 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:18:48.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:18:48.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:18:48.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:18:48.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:18:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:18:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:18:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:18:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:18:48.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:18:48.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:18:48.759 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:18:48.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:18:48.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:48.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:18:48.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=450971566096 2024-06-11T18:18:48.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 450971566096 2024-06-11T18:18:48.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566096' 2024-06-11T18:18:48.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:18:48.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:18:49.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=549755813890 2024-06-11T18:18:49.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 549755813890 2024-06-11T18:18:49.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-450971566096 1-549755813890' 2024-06-11T18:18:49.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:49.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-450971566096 2024-06-11T18:18:49.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:49.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:18:49.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-450971566096 2024-06-11T18:18:49.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=450971566096 2024-06-11T18:18:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 450971566096' 2024-06-11T18:18:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 450971566096 2024-06-11T18:18:49.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:49.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566095 -lt 450971566096 2024-06-11T18:18:49.542 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:18:50.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:18:50.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:18:50.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 450971566096 -lt 450971566096 2024-06-11T18:18:50.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:18:50.882 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-549755813890 2024-06-11T18:18:50.882 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:18:50.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:18:50.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-549755813890 2024-06-11T18:18:50.885 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:18:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=549755813890 2024-06-11T18:18:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2024-06-11T18:18:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 549755813890 2024-06-11T18:18:50.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:18:51.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 549755813890 -lt 549755813890 2024-06-11T18:18:51.275 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:18:51.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:51.276 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:51.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:18:51.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:51.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:51.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:51.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:51.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:51.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:51.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:52.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T18:18:52.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:52.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:52.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:52.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-06-11T18:18:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-11T18:18:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:18:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-11T18:18:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:18:52.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:18:52.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:18:52.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:52.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:52.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:52.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:52.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:52.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:52.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-11T18:18:52.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:52.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:52.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:18:53.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:18:53.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:18:53.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:18:53.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:18:53.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:18:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:18:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:18:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:18:53.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:18:53.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:18:53.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:18:53.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:18:54.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:18:54.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:18:54.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:18:54.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:18:54.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:18:54.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:18:54.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:18:54.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-11T18:18:54.657 INFO:tasks.workunit.client.0.smithi012.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-11T18:18:54.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:54.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-11T18:18:54.658 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-06-11T18:18:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:18:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:54.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-11T18:18:54.660 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-06-11T18:18:54.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:18:54.660 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:54.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:54.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-11T18:18:54.661 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-06-11T18:18:54.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:18:54.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:54.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:54.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-11T18:18:54.663 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-06-11T18:18:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:18:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:18:54.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-11T18:18:54.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:18:54.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:18:54.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:18:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:18:58.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:18:58.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:18:58.025 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:18:58.026 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:18:58.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:18:58.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:18:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:18:58.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:18:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:18:58.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:18:58.063 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:58.062+0000 7f489ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:58.066 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:58.066+0000 7f489ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:58.067 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:58.066+0000 7f489ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:18:58.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:18:58.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:58.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:18:58.613 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:18:58.610+0000 7f489ba017c0 -1 Falling back to public interface 2024-06-11T18:18:58.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:18:59.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:18:59.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:18:59.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:18:59.753 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:18:59.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:18:59.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:00.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:01.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:01.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:01.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:19:01.789 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:19:01.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:01.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:01.951 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:01.950+0000 7f489ba017c0 -1 osd.0 130 log_to_monitors true 2024-06-11T18:19:02.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:03.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:03.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:19:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:19:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:03.662 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [105,130) [v2:127.0.0.1:6802/2521847588,v1:127.0.0.1:6803/2521847588] [v2:127.0.0.1:6804/2521847588,v1:127.0.0.1:6805/2521847588] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:19:03.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:19:03.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:19:03.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:19:03.664 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:19:03.664 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:19:03.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:19:03.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:19:03.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:19:03.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:19:03.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:19:03.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:19:03.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:19:03.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:19:03.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:19:04.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:19:04.126 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:19:04.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:19:04.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:04.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:19:04.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=575525617666 2024-06-11T18:19:04.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 575525617666 2024-06-11T18:19:04.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617666' 2024-06-11T18:19:04.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:04.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:19:04.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=549755813894 2024-06-11T18:19:04.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 549755813894 2024-06-11T18:19:04.482 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617666 1-549755813894' 2024-06-11T18:19:04.483 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:04.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-575525617666 2024-06-11T18:19:04.483 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:04.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:19:04.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-575525617666 2024-06-11T18:19:04.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:04.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=575525617666 2024-06-11T18:19:04.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2024-06-11T18:19:04.486 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 575525617666 2024-06-11T18:19:04.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:04.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 575525617666 2024-06-11T18:19:04.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:19:05.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:19:05.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 575525617666 -lt 575525617666 2024-06-11T18:19:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:06.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-549755813894 2024-06-11T18:19:06.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:06.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:19:06.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-549755813894 2024-06-11T18:19:06.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:06.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=549755813894 2024-06-11T18:19:06.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 549755813894' 2024-06-11T18:19:06.134 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 549755813894 2024-06-11T18:19:06.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:19:06.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 549755813894 -lt 549755813894 2024-06-11T18:19:06.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:19:06.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:06.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:06.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:19:06.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:19:06.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:19:06.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:19:06.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:19:06.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:19:06.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:19:06.890 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:19:07.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:19:07.256 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:19:07.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:07.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:19:07.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:19:07.651 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:19:07.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:19:07.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:19:07.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:19:08.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:08.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-06-11T18:19:10.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:10.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:19:10.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:19:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:19:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:19:10.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:19:10.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:19:10.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:19:10.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:19:10.977 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:10.974+0000 7ffa593927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:10.980 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:10.978+0000 7ffa593927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:10.981 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:10.978+0000 7ffa593927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:11.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:11.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:11.524 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:11.522+0000 7ffa593927c0 -1 Falling back to public interface 2024-06-11T18:19:11.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:19:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:19:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:12.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:13.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:13.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:13.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:19:13.945 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:19:13.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:13.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:14.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:14.335 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:14.334+0000 7ffa593927c0 -1 osd.1 135 log_to_monitors true 2024-06-11T18:19:15.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:15.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:15.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:19:15.283 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:19:15.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:15.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 139 up_thru 139 down_at 136 last_clean_interval [128,135) [v2:127.0.0.1:6810/3399748921,v1:127.0.0.1:6811/3399748921] [v2:127.0.0.1:6812/3399748921,v1:127.0.0.1:6813/3399748921] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:19:15.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:19:15.848 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:19:15.848 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:19:15.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:19:15.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:19:15.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:19:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:19:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:19:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:19:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:19:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:19:15.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:19:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:19:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:19:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:19:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:16.314 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:19:16.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=575525617669 2024-06-11T18:19:16.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 575525617669 2024-06-11T18:19:16.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617669' 2024-06-11T18:19:16.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:16.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:19:16.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=597000454146 2024-06-11T18:19:16.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 597000454146 2024-06-11T18:19:16.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617669 1-597000454146' 2024-06-11T18:19:16.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:16.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-575525617669 2024-06-11T18:19:16.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:16.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:19:16.708 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-575525617669 2024-06-11T18:19:16.709 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:16.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=575525617669 2024-06-11T18:19:16.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 575525617669' 2024-06-11T18:19:16.710 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 575525617669 2024-06-11T18:19:16.711 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:17.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 575525617668 -lt 575525617669 2024-06-11T18:19:17.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:19:18.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:19:18.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:18.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 575525617669 -lt 575525617669 2024-06-11T18:19:18.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:18.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:18.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-597000454146 2024-06-11T18:19:18.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:19:18.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-597000454146 2024-06-11T18:19:18.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:18.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=597000454146 2024-06-11T18:19:18.377 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2024-06-11T18:19:18.377 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 597000454146 2024-06-11T18:19:18.377 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:19:18.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 597000454146 -lt 597000454146 2024-06-11T18:19:18.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:19:18.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:18.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:19.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:19:19.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:19:19.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:19:19.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:19:19.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:19:19.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:19:19.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:19:19.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:19:19.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:19:19.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:19:19.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:19.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:19:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:19:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:19:19.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-06-11T18:19:19.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:19:19.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-11T18:19:19.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-06-11T18:19:19.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:19:19.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:19:19.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-11T18:19:19.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:19:19.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:19:19.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:21.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-06-11T18:19:22.505 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:19:23.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:19:23.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:19:23.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:19:23.045 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:19:23.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:19:23.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:19:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:19:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:19:23.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:19:23.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:19:23.081 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:23.077+0000 7f90628a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:23.084 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:23.081+0000 7f90628a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:23.086 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:23.081+0000 7f90628a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:23.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:19:23.393 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:19:23.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:23.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:23.625 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:23.621+0000 7f90628a67c0 -1 Falling back to public interface 2024-06-11T18:19:23.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:24.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:24.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:24.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:19:24.729 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:19:24.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:24.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:25.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:26.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:26.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:26.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:19:26.063 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:19:26.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:26.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:26.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:26.599 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:26.597+0000 7f90628a67c0 -1 osd.1 140 log_to_monitors true 2024-06-11T18:19:27.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:27.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:27.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:19:27.404 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:19:27.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:27.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [139,140) [v2:127.0.0.1:6810/3942990666,v1:127.0.0.1:6811/3942990666] [v2:127.0.0.1:6812/3942990666,v1:127.0.0.1:6813/3942990666] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:19:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:19:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:19:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:19:27.794 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:19:27.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:19:27.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:19:27.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:19:27.917 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:19:28.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:19:28.357 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:19:28.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:19:28.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:28.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:19:28.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=575525617673 2024-06-11T18:19:28.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 575525617673 2024-06-11T18:19:28.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617673' 2024-06-11T18:19:28.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:28.513 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:19:28.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=618475290626 2024-06-11T18:19:28.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 618475290626 2024-06-11T18:19:28.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-575525617673 1-618475290626' 2024-06-11T18:19:28.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:28.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-575525617673 2024-06-11T18:19:28.669 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:28.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:19:28.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-575525617673 2024-06-11T18:19:28.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:28.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=575525617673 2024-06-11T18:19:28.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 575525617673' 2024-06-11T18:19:28.672 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 575525617673 2024-06-11T18:19:28.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:28.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 575525617671 -lt 575525617673 2024-06-11T18:19:28.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:19:29.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:19:29.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:30.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 575525617673 -lt 575525617673 2024-06-11T18:19:30.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:30.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:30.327 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-618475290626 2024-06-11T18:19:30.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:19:30.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-618475290626 2024-06-11T18:19:30.329 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:30.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=618475290626 2024-06-11T18:19:30.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-06-11T18:19:30.331 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 618475290626 2024-06-11T18:19:30.331 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:19:30.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 618475290626 -lt 618475290626 2024-06-11T18:19:30.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:19:30.702 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:30.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:19:31.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:19:31.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:19:31.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:19:31.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:19:31.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:19:31.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:19:31.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:19:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:19:31.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:19:31.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:31.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:31.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:19:31.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:19:31.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:19:31.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:19:31.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-11T18:19:31.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-06-11T18:19:31.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:19:31.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:19:31.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:19:31.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:19:31.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:19:31.905 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:19:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:19:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:19:31.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:19:33.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:19:33.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:19:33.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-06-11T18:19:34.557 INFO:tasks.workunit.client.0.smithi012.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-11T18:19:35.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:19:35.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:19:35.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:19:35.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:19:35.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:19:35.343 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:19:35.344 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:19:35.344 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:19:35.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:19:35.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:19:35.346 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:19:35.346 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:19:35.347 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:19:35.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:19:35.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:19:35.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:19:35.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:19:35.381 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:35.377+0000 7f55e74cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:35.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:35.381+0000 7f55e74cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:35.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:35.381+0000 7f55e74cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:35.920 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:35.917+0000 7f55e74cc7c0 -1 Falling back to public interface 2024-06-11T18:19:36.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:37.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:37.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:37.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:19:37.062 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:19:37.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:37.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:37.398 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:38.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:38.632 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:38.629+0000 7f55e74cc7c0 -1 osd.0 145 log_to_monitors true 2024-06-11T18:19:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:39.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:39.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:39.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:19:39.791 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:19:39.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:39.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [134,145) [v2:127.0.0.1:6802/650250129,v1:127.0.0.1:6803/650250129] [v2:127.0.0.1:6804/650250129,v1:127.0.0.1:6805/650250129] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:19:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:19:40.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:19:40.133 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:19:40.133 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:19:40.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:19:40.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:19:40.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:19:40.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:19:40.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:19:40.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:19:40.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:19:40.264 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:19:40.264 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:19:40.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:19:40.723 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:19:40.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:19:40.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:40.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:19:40.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=639950127106 2024-06-11T18:19:40.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 639950127106 2024-06-11T18:19:40.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-639950127106' 2024-06-11T18:19:40.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:40.875 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:19:41.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=618475290629 2024-06-11T18:19:41.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 618475290629 2024-06-11T18:19:41.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-06-11T18:19:41.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:41.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-639950127106 2024-06-11T18:19:41.020 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:41.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:19:41.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-639950127106 2024-06-11T18:19:41.022 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:41.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=639950127106 2024-06-11T18:19:41.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-06-11T18:19:41.023 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 639950127106 2024-06-11T18:19:41.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:41.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 639950127106 2024-06-11T18:19:41.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:19:42.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:19:42.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:42.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 639950127106 -lt 639950127106 2024-06-11T18:19:42.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:42.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-618475290629 2024-06-11T18:19:42.732 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:42.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:19:42.735 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-618475290629 2024-06-11T18:19:42.735 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:42.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=618475290629 2024-06-11T18:19:42.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-06-11T18:19:42.737 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 618475290629 2024-06-11T18:19:42.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:19:43.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 618475290629 -lt 618475290629 2024-06-11T18:19:43.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:19:43.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:43.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:19:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:19:43.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:19:43.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:19:43.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:19:43.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:19:43.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:19:43.571 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:19:43.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:19:43.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:19:43.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:43.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:44.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:19:44.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:19:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:19:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:19:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:19:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-11T18:19:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:19:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:19:45.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:19:45.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:19:45.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:45.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-11T18:19:46.945 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:19:46.945 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:19:47.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:19:47.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:19:47.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:19:47.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:19:47.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:19:47.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:19:47.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:19:47.496 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:19:47.496 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:19:47.496 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:19:47.497 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:19:47.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:19:47.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:19:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:19:47.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:19:47.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:19:47.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:19:47.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:19:47.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:19:47.535 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:47.533+0000 7f68930cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:47.542 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:47.541+0000 7f68930cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:47.544 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:47.541+0000 7f68930cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:47.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:19:47.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:19:47.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:48.092 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:48.089+0000 7f68930cb7c0 -1 Falling back to public interface 2024-06-11T18:19:48.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:49.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:49.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:50.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:50.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:50.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:19:50.592 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:19:50.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:50.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:50.848 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:50.845+0000 7f68930cb7c0 -1 osd.1 150 log_to_monitors true 2024-06-11T18:19:50.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:19:51.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:19:51.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:51.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:19:51.936 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:19:51.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:51.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:19:52.439 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/2548204128,v1:127.0.0.1:6811/2548204128] [v2:127.0.0.1:6812/2548204128,v1:127.0.0.1:6813/2548204128] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:19:52.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:19:52.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:19:52.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:19:52.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:19:52.440 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:19:52.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:19:52.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:19:52.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:19:52.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:19:52.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:19:52.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:19:52.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:19:52.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:19:52.565 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:19:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:19:52.940 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:19:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:19:52.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:52.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:19:53.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=639950127109 2024-06-11T18:19:53.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 639950127109 2024-06-11T18:19:53.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-639950127109' 2024-06-11T18:19:53.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:19:53.134 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:19:53.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=661424963586 2024-06-11T18:19:53.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 661424963586 2024-06-11T18:19:53.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-06-11T18:19:53.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:53.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:53.289 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-639950127109 2024-06-11T18:19:53.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:19:53.290 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-639950127109 2024-06-11T18:19:53.291 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:53.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=639950127109 2024-06-11T18:19:53.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-06-11T18:19:53.292 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 639950127109 2024-06-11T18:19:53.292 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:19:53.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 639950127109 -lt 639950127109 2024-06-11T18:19:53.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:19:53.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-661424963586 2024-06-11T18:19:53.614 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:19:53.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:19:53.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-661424963586 2024-06-11T18:19:53.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:19:53.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=661424963586 2024-06-11T18:19:53.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-06-11T18:19:53.619 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 661424963586 2024-06-11T18:19:53.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:19:53.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 661424963586 -lt 661424963586 2024-06-11T18:19:53.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:19:53.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:53.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:54.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:19:54.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:19:54.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:19:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:19:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:19:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:19:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:19:54.372 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:19:54.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:19:54.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:19:54.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:19:54.699 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:19:55.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:19:55.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:19:55.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:19:55.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:19:55.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-11T18:19:55.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:19:55.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:19:55.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:19:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:19:58.177 INFO:tasks.workunit.client.0.smithi012.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:19:59.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:19:59.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:19:59.084 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:19:59.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:19:59.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:19:59.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:19:59.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:19:59.088 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:19:59.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:19:59.088 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:19:59.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:19:59.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:19:59.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:19:59.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:19:59.123 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:59.121+0000 7fe17e2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:59.125 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:59.121+0000 7fe17e2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:59.127 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:59.125+0000 7fe17e2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:19:59.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:19:59.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:19:59.669 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:19:59.665+0000 7fe17e2bc7c0 -1 Falling back to public interface 2024-06-11T18:19:59.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:00.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:00.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:00.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:20:00.775 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:20:00.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:00.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:01.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:02.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:02.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:02.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:20:02.115 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:20:02.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:02.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:02.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:02.567 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:02.565+0000 7fe17e2bc7c0 -1 osd.0 155 log_to_monitors true 2024-06-11T18:20:03.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:20:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:20:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:03.929 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 159 up_thru 154 down_at 156 last_clean_interval [149,155) [v2:127.0.0.1:6802/726578952,v1:127.0.0.1:6803/726578952] [v2:127.0.0.1:6804/726578952,v1:127.0.0.1:6805/726578952] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:20:03.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:20:03.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:20:04.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:20:04.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:20:04.579 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:20:04.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:20:04.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:04.580 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:20:04.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=682899800066 2024-06-11T18:20:04.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 682899800066 2024-06-11T18:20:04.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-682899800066' 2024-06-11T18:20:04.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:04.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:20:05.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=661424963589 2024-06-11T18:20:05.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 661424963589 2024-06-11T18:20:05.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-682899800066 1-661424963589' 2024-06-11T18:20:05.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:05.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-682899800066 2024-06-11T18:20:05.227 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:05.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:20:05.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-682899800066 2024-06-11T18:20:05.230 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:05.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=682899800066 2024-06-11T18:20:05.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 682899800066' 2024-06-11T18:20:05.232 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 682899800066 2024-06-11T18:20:05.233 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 682899800066 2024-06-11T18:20:05.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:20:06.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:20:06.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:06.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 682899800066 -lt 682899800066 2024-06-11T18:20:06.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:06.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-661424963589 2024-06-11T18:20:06.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:06.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:20:06.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:06.992 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-661424963589 2024-06-11T18:20:06.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=661424963589 2024-06-11T18:20:06.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-06-11T18:20:06.993 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 661424963589 2024-06-11T18:20:06.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 661424963589 -lt 661424963589 2024-06-11T18:20:07.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:20:07.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:07.353 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:07.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:20:07.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:07.751 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:08.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:20:08.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:08.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:08.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:08.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:20:08.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:20:08.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:20:08.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:20:08.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:20:08.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:20:08.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:20:08.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:08.575 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:08.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:20:08.908 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:08.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:08.909 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:09.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:20:09.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:20:09.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:20:09.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:20:09.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:20:09.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:20:09.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:20:09.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:20:09.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:20:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:20:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:20:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:20:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:20:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:20:09.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:20:09.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:09.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:09.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:09.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:09.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:09.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:09.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:10.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:20:10.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:10.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:10.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:20:10.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:20:10.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:12.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-11T18:20:13.240 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:20:13.240 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:13.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:20:13.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:13.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:20:13.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:20:13.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:20:13.815 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:20:13.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:20:13.815 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:20:13.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:20:13.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:20:13.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:20:13.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:20:13.851 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:13.849+0000 7f610751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:13.854 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:13.849+0000 7f610751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:13.855 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:13.853+0000 7f610751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:14.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:20:14.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:20:14.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:20:14.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:20:14.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:20:14.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:14.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:20:14.157 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:20:14.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:14.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:14.393 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:14.389+0000 7f610751e7c0 -1 Falling back to public interface 2024-06-11T18:20:14.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:20:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:20:15.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:15.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:15.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:16.750 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:16.745+0000 7f610751e7c0 -1 osd.1 161 log_to_monitors true 2024-06-11T18:20:16.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:16.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:16.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:20:16.862 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:20:16.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:16.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:17.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:18.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:18.646 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/294264197,v1:127.0.0.1:6811/294264197] [v2:127.0.0.1:6812/294264197,v1:127.0.0.1:6813/294264197] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:20:18.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:20:18.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:20:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:20:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:20:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:20:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:20:18.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:20:18.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:20:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:20:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:20:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:20:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:19.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:20:19.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=682899800070 2024-06-11T18:20:19.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 682899800070 2024-06-11T18:20:19.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-682899800070' 2024-06-11T18:20:19.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:19.440 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:20:19.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=708669603842 2024-06-11T18:20:19.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 708669603842 2024-06-11T18:20:19.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-682899800070 1-708669603842' 2024-06-11T18:20:19.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:19.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-682899800070 2024-06-11T18:20:19.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:19.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:20:19.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-682899800070 2024-06-11T18:20:19.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:19.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=682899800070 2024-06-11T18:20:19.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 682899800070' 2024-06-11T18:20:19.625 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 682899800070 2024-06-11T18:20:19.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:19.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 682899800069 -lt 682899800070 2024-06-11T18:20:19.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:20:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:20:20.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:21.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 682899800069 -lt 682899800070 2024-06-11T18:20:21.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:20:22.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:20:22.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:22.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 682899800070 -lt 682899800070 2024-06-11T18:20:22.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:22.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-708669603842 2024-06-11T18:20:22.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:22.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:20:22.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-708669603842 2024-06-11T18:20:22.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:22.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=708669603842 2024-06-11T18:20:22.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-06-11T18:20:22.615 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 708669603842 2024-06-11T18:20:22.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:22.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 708669603842 -lt 708669603842 2024-06-11T18:20:22.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:20:22.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:22.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:23.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:20:23.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:23.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:23.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:23.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:23.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:23.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:23.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:23.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:20:23.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:23.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:23.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:24.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:20:24.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:20:24.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:20:24.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:20:24.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-11T18:20:24.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-06-11T18:20:24.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-11T18:20:24.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:20:24.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:20:24.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:20:24.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:20:24.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:20:24.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:20:24.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:20:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:26.787 INFO:tasks.workunit.client.0.smithi012.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:20:27.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:20:27.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:20:27.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:20:27.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:20:27.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:20:27.707 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:27.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:27.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:20:27.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:20:27.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:20:27.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:20:27.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:20:27.710 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:20:27.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:20:27.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:20:27.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:20:27.711 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:20:27.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:20:27.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:20:27.750 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:27.745+0000 7f2a921b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:27.753 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:27.749+0000 7f2a921b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:27.755 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:27.753+0000 7f2a921b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:20:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:28.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:28.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:28.293+0000 7f2a921b27c0 -1 Falling back to public interface 2024-06-11T18:20:28.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:20:29.419 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:20:29.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:29.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:29.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:30.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:30.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:30.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:20:30.782 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:20:30.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:30.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:31.073 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:31.069+0000 7f2a921b27c0 -1 osd.0 167 log_to_monitors true 2024-06-11T18:20:31.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:32.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:32.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:32.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:20:32.115 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:20:32.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:32.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 171 up_thru 166 down_at 168 last_clean_interval [159,167) [v2:127.0.0.1:6802/858191522,v1:127.0.0.1:6803/858191522] [v2:127.0.0.1:6804/858191522,v1:127.0.0.1:6805/858191522] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:20:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:20:32.558 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:20:32.558 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:20:32.558 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:20:32.559 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:20:32.559 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:20:32.559 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:20:32.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:20:32.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:20:32.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:20:32.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:20:32.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:20:32.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:20:33.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:20:33.013 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:20:33.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:20:33.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:33.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:20:33.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=734439407618 2024-06-11T18:20:33.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 734439407618 2024-06-11T18:20:33.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407618' 2024-06-11T18:20:33.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:33.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=708669603846 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 708669603846 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407618 1-708669603846' 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-734439407618 2024-06-11T18:20:33.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:33.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:20:33.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-734439407618 2024-06-11T18:20:33.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:33.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=734439407618 2024-06-11T18:20:33.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-06-11T18:20:33.415 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 734439407618 2024-06-11T18:20:33.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:33.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 734439407618 -lt 734439407618 2024-06-11T18:20:33.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:33.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-708669603846 2024-06-11T18:20:33.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:33.903 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:20:33.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-708669603846 2024-06-11T18:20:33.904 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:33.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=708669603846 2024-06-11T18:20:33.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 708669603846' 2024-06-11T18:20:33.906 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 708669603846 2024-06-11T18:20:33.906 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:34.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 708669603845 -lt 708669603846 2024-06-11T18:20:34.285 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:20:35.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:20:35.287 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:35.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 708669603846 -lt 708669603846 2024-06-11T18:20:35.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:20:35.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:35.623 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:36.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:20:36.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:36.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:36.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:36.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:36.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:36.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:36.026 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:36.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:20:36.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:36.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:36.382 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:20:36.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:20:36.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:20:36.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:36.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:37.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:20:37.300 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:37.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:37.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:37.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:20:37.700 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:20:37.701 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:20:38.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:20:38.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:20:38.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:20:38.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:20:38.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:20:38.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:20:38.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:38.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:38.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:38.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:38.338 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:38.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:38.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:38.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:20:38.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:38.671 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:38.671 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:20:39.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:20:39.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:20:40.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:20:40.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:40.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:20:40.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:20:40.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:20:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:20:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-11T18:20:41.045 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:20:41.045 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:20:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:20:41.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:20:41.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:20:41.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:20:41.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:20:41.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:20:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:20:41.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:20:41.614 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:20:41.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:20:41.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:20:41.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:20:41.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:20:41.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:20:41.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:20:41.650 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:41.645+0000 7f3aa72447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:41.652 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:41.649+0000 7f3aa72447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:41.654 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:41.653+0000 7f3aa72447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:42.196 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:42.193+0000 7f3aa72447c0 -1 Falling back to public interface 2024-06-11T18:20:42.334 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:43.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:20:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:20:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:43.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:44.581 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:44.577+0000 7f3aa72447c0 -1 osd.1 173 log_to_monitors true 2024-06-11T18:20:44.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:20:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:20:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:45.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:46.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:20:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:20:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 177 up_thru 177 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/767837093,v1:127.0.0.1:6811/767837093] [v2:127.0.0.1:6812/767837093,v1:127.0.0.1:6813/767837093] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:20:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:20:46.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:20:46.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:20:46.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:20:46.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:20:46.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:20:46.938 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:20:46.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:20:46.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:46.939 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:20:47.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=734439407622 2024-06-11T18:20:47.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 734439407622 2024-06-11T18:20:47.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407622' 2024-06-11T18:20:47.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:47.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:20:47.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=760209211394 2024-06-11T18:20:47.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 760209211394 2024-06-11T18:20:47.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407622 1-760209211394' 2024-06-11T18:20:47.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:47.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-734439407622 2024-06-11T18:20:47.235 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:47.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:20:47.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-734439407622 2024-06-11T18:20:47.237 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:47.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=734439407622 2024-06-11T18:20:47.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 734439407622' 2024-06-11T18:20:47.238 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 734439407622 2024-06-11T18:20:47.238 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 734439407622 -lt 734439407622 2024-06-11T18:20:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:47.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-760209211394 2024-06-11T18:20:47.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:47.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:20:47.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-760209211394 2024-06-11T18:20:47.615 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:47.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=760209211394 2024-06-11T18:20:47.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-06-11T18:20:47.617 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 760209211394 2024-06-11T18:20:47.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:47.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 760209211394 -lt 760209211394 2024-06-11T18:20:47.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:20:47.934 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:47.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:48.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:20:48.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:48.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:48.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:48.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:48.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:48.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:48.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:20:48.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:20:48.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:20:48.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:48.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:20:49.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:20:49.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:20:50.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:20:50.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-11T18:20:50.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:20:50.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:20:50.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:20:50.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:20:50.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:50.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-06-11T18:20:51.080 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:20:51.080 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:20:51.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:20:51.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:20:51.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:20:51.623 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:20:51.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:20:51.671 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:51.669+0000 7f6049a367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:51.674 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:51.669+0000 7f6049a367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:51.675 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:51.673+0000 7f6049a367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:20:51.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:20:51.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:20:51.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:20:51.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:51.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:52.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:52.229+0000 7f6049a367c0 -1 Falling back to public interface 2024-06-11T18:20:52.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:53.267 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:53.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:53.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:20:53.268 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:20:53.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:53.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:53.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:20:54.649 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:20:54.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:54.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:54.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:20:54.988 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:20:54.985+0000 7f6049a367c0 -1 osd.1 179 log_to_monitors true 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:20:55.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 183 up_thru 183 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/836103378,v1:127.0.0.1:6811/836103378] [v2:127.0.0.1:6812/836103378,v1:127.0.0.1:6813/836103378] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:20:56.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:20:56.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:20:56.368 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:20:56.368 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:20:56.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:20:56.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:20:56.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:20:56.485 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:20:56.951 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:20:56.952 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:20:56.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:20:56.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:56.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:20:57.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=734439407625 2024-06-11T18:20:57.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 734439407625 2024-06-11T18:20:57.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407625' 2024-06-11T18:20:57.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:20:57.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:20:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=785979015170 2024-06-11T18:20:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 785979015170 2024-06-11T18:20:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-734439407625 1-785979015170' 2024-06-11T18:20:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:57.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-734439407625 2024-06-11T18:20:57.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:20:57.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-734439407625 2024-06-11T18:20:57.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=734439407625 2024-06-11T18:20:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 734439407625' 2024-06-11T18:20:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 734439407625 2024-06-11T18:20:57.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:20:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 734439407625 -lt 734439407625 2024-06-11T18:20:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:20:57.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-785979015170 2024-06-11T18:20:57.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:20:57.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:20:57.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-785979015170 2024-06-11T18:20:57.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:20:57.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=785979015170 2024-06-11T18:20:57.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-06-11T18:20:57.842 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 785979015170 2024-06-11T18:20:57.842 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:58.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 785979015170 2024-06-11T18:20:58.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:20:59.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:20:59.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:20:59.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 785979015170 -lt 785979015170 2024-06-11T18:20:59.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:20:59.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:20:59.532 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:20:59.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:20:59.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:20:59.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:20:59.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:20:59.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:20:59.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:20:59.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:20:59.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:00.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:00.273 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:00.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:00.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:00.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:21:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:21:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:21:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:00.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:21:00.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:00.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:00.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:00.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:00.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:00.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:00.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:00.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:01.084 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:01.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:01.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:01.086 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:01.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:01.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:21:01.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:21:01.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:21:01.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:21:01.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:21:01.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:21:01.504 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:21:01.504 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:21:01.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:21:01.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:21:01.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:21:01.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:01.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:21:02.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:02.083 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:02.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:02.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:02.084 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:02.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:02.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:02.085 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:02.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:02.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:02.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:02.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:02.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:21:02.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:21:02.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:21:03.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:21:03.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:21:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T18:21:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T18:21:03.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:03.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:03.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:03.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:03.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:03.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:03.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:03.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:03.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:21:03.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:03.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:03.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:04.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:21:04.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:21:04.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:21:04.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-11T18:21:04.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-11T18:21:04.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-06-11T18:21:04.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-11T18:21:04.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-11T18:21:04.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-11T18:21:04.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-11T18:21:04.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-11T18:21:04.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:21:04.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-11T18:21:04.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-11T18:21:04.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-06-11T18:21:04.947 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: {} 2024-06-11T18:21:04.947 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-11T18:21:04.965 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: {} 2024-06-11T18:21:04.965 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:21:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ11 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ11 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:21:04.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-11T18:21:04.984 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-11T18:21:04.984 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:21:05.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:21:05.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:21:05.335 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T18:21:05.335 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:21:05.335 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:21:05.335 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:21:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:21:05.336 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:21:05.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:21:05.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:21:05.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:21:05.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-06-11T18:21:05.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:21:05.455 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:21:05.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:05.455 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:05.456 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:05.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:21:05.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:21:05.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-11T18:21:05.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-11T18:21:05.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=mdata 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ12 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:21:05.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:21:05.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-06-11T18:21:05.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:21:05.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ12 2024-06-11T18:21:05.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:21:05.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-11T18:21:05.588 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:21:05.588 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-06-11T18:21:05.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:21:05.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:21:05.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:21:05.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:21:05.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:21:05.921 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-11T18:21:05.921 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:21:05.921 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:21:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:05.922 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:21:05.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:21:06.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:21:06.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:21:06.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:06.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:21:06.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:21:06.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=mdata 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:21:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ13 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-06-11T18:21:06.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:21:06.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ13 2024-06-11T18:21:06.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:21:06.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-11T18:21:06.175 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-11T18:21:06.175 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:21:06.507 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-11T18:21:06.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:21:06.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-11T18:21:06.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:21:06.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-11T18:21:06.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:21:06.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:21:06.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:21:06.511 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-11T18:21:06.511 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:21:06.511 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:21:06.512 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:06.512 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:06.512 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:06.512 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:21:06.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:21:06.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:21:06.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:21:06.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:21:06.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-06-11T18:21:06.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:21:06.635 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:21:06.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:21:06.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ13 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ13 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-11T18:21:06.756 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-11T18:21:06.757 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:21:07.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:21:07.090 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-11T18:21:07.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:21:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-11T18:21:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:21:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T18:21:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:21:07.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:21:07.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:21:07.092 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T18:21:07.092 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:21:07.092 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:21:07.093 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:07.093 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:07.094 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:07.094 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:21:07.094 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:21:07.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:21:07.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:21:07.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:21:07.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-06-11T18:21:07.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:21:07.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:21:07.222 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:21:07.223 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:07.223 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:07.223 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:21:07.223 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:21:07.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-11T18:21:07.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-06-11T18:21:07.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=3.0 2024-06-11T18:21:07.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 3.0 2024-06-11T18:21:07.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:21:07.357 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:21:07.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:21:07.358 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:21:07.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:07.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 3.0 2024-06-11T18:21:07.995 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 3.0 2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:21:08.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:21:08.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:21:08.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:21:08.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:21:08.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:21:08.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:21:08.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:08.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:21:09.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:21:09.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:21:09.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:21:09.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:21:09.339 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:21:09.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:21:09.340 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:21:09.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:09.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:21:10.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:21:10.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:21:10.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:21:10.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:21:10.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:21:10.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:21:10.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:21:11.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:11.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:21:12.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:21:12.022 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:21:12.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:21:12.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:21:12.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:21:12.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:21:12.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:21:12.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:21:08.011152+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-11T18:21:12.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-11T18:21:12.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-06-11T18:21:12.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-06-11T18:21:12.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-06-11T18:21:12.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:21:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-06-11T18:21:12.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-11T18:21:12.624 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:21:12.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-11T18:21:12.654 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:21:12.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-11T18:21:12.683 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-11T18:21:12.713 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:21:12.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-06-11T18:21:12.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-11T18:21:12.743 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:21:12.743 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:21:12.743 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-11T18:21:12.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:21:12.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-11T18:21:12.791 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:21:12.791 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:21:12.791 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:21:12.791 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-11T18:21:12.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:21:12.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:21:12.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:21:12.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-06-11T18:21:12.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-11T18:21:12.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-11T18:21:12.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-06-11T18:21:12.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:21:12.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:21:12.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:21:12.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:21:12.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:21:12.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:21:12.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:21:13.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-11T18:21:15.019 INFO:tasks.workunit.client.0.smithi012.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-11T18:21:15.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:21:15.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:21:15.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:21:15.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:21:15.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:21:15.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:15.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:21:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:21:15.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:21:15.555 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:21:15.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:21:15.555 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:21:15.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:21:15.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:21:15.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:21:15.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:21:15.590 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:15.585+0000 7f7d0cfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:15.593 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:15.589+0000 7f7d0cfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:15.594 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:15.593+0000 7f7d0cfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:15.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:16.137 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:16.133+0000 7f7d0cfa57c0 -1 Falling back to public interface 2024-06-11T18:21:16.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:17.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:17.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:21:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:21:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:17.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:17.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:18.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:18.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:18.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:21:18.543 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:21:18.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:18.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:18.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:18.958 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:18.953+0000 7f7d0cfa57c0 -1 osd.0 185 log_to_monitors true 2024-06-11T18:21:19.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:19.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:19.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:21:19.882 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:21:19.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:19.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:20.245 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 189 up_thru 183 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/4159629653,v1:127.0.0.1:6803/4159629653] [v2:127.0.0.1:6804/4159629653,v1:127.0.0.1:6805/4159629653] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:21:20.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:21:20.246 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:21:20.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:21:20.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:21:20.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:21:20.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:21:20.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:21:20.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:21:20.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:21:20.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:21:20.763 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:21:20.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:21:20.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:20.764 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:21:21.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=811748818946 2024-06-11T18:21:21.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 811748818946 2024-06-11T18:21:21.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-811748818946' 2024-06-11T18:21:21.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:21.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:21:21.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=785979015176 2024-06-11T18:21:21.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 785979015176 2024-06-11T18:21:21.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-811748818946 1-785979015176' 2024-06-11T18:21:21.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:21.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-811748818946 2024-06-11T18:21:21.390 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:21.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:21:21.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-811748818946 2024-06-11T18:21:21.393 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:21.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=811748818946 2024-06-11T18:21:21.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-06-11T18:21:21.395 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 811748818946 2024-06-11T18:21:21.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:21:21.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 811748818946 -lt 811748818946 2024-06-11T18:21:21.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:21.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-785979015176 2024-06-11T18:21:21.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:21.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:21:21.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-785979015176 2024-06-11T18:21:21.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:21.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=785979015176 2024-06-11T18:21:21.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 785979015176' 2024-06-11T18:21:21.965 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 785979015176 2024-06-11T18:21:21.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:21:22.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 785979015175 -lt 785979015176 2024-06-11T18:21:22.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:21:23.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:21:23.351 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:21:23.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 785979015176 -lt 785979015176 2024-06-11T18:21:23.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:21:23.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:23.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:24.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:21:24.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:24.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:24.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:24.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:24.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:24.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:24.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:24.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:21:24.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:24.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:24.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:24.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:24.867 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:24.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:24.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:24.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:25.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:25.196 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:25.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:25.197 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:25.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:25.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:21:25.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:21:25.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:21:25.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:21:25.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:21:25.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:25.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:21:26.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:26.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:26.200 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:26.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:26.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:26.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:26.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:26.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:26.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:21:26.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:26.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:26.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:21:26.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:21:26.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:21:26.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-11T18:21:26.912 INFO:tasks.workunit.client.0.smithi012.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-11T18:21:26.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-11T18:21:26.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:21:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:21:26.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:21:26.914 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:21:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:21:28.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:21:28.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-06-11T18:21:28.906 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:21:28.907 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:21:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:21:29.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:21:29.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:21:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:21:29.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:21:29.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:21:29.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:21:29.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:21:29.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:21:29.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:21:29.485 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:29.480+0000 7f67e6a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:29.492 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:29.488+0000 7f67e6a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:29.494 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:29.492+0000 7f67e6a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:29.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:21:29.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:21:29.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:21:29.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:29.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:30.053 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:30.048+0000 7f67e6a857c0 -1 Falling back to public interface 2024-06-11T18:21:30.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:31.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:31.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:31.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:21:31.117 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:21:31.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:31.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:31.454 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:32.446 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:32.440+0000 7f67e6a857c0 -1 osd.1 191 log_to_monitors true 2024-06-11T18:21:32.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:32.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:32.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:21:32.456 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:21:32.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:32.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:32.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:33.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:33.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:33.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:21:33.788 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:21:33.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:33.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:34.127 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 195 up_thru 195 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/640397339,v1:127.0.0.1:6811/640397339] [v2:127.0.0.1:6812/640397339,v1:127.0.0.1:6813/640397339] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:21:34.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:21:34.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:21:34.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:21:34.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:21:34.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:21:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:21:34.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:21:34.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:21:34.130 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:21:34.130 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:21:34.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:21:34.130 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:21:34.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:21:34.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:21:34.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:21:34.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:21:34.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:21:34.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:21:34.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:21:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:21:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:21:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:21:34.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:34.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:21:34.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=811748818950 2024-06-11T18:21:34.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 811748818950 2024-06-11T18:21:34.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-811748818950' 2024-06-11T18:21:34.831 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:34.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:21:34.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=837518622722 2024-06-11T18:21:34.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 837518622722 2024-06-11T18:21:34.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-811748818950 1-837518622722' 2024-06-11T18:21:34.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:34.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-811748818950 2024-06-11T18:21:34.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:34.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:21:34.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-811748818950 2024-06-11T18:21:34.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:34.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=811748818950 2024-06-11T18:21:34.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 811748818950' 2024-06-11T18:21:34.981 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 811748818950 2024-06-11T18:21:34.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:21:35.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 811748818948 -lt 811748818950 2024-06-11T18:21:35.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:21:36.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:21:36.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:21:36.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 811748818950 -lt 811748818950 2024-06-11T18:21:36.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:36.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-837518622722 2024-06-11T18:21:36.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:36.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:21:36.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-837518622722 2024-06-11T18:21:36.644 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:36.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=837518622722 2024-06-11T18:21:36.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-06-11T18:21:36.646 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 837518622722 2024-06-11T18:21:36.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:21:37.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 837518622722 -lt 837518622722 2024-06-11T18:21:37.007 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:21:37.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:37.008 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:37.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:37.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:37.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:21:37.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:37.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:37.818 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:38.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:21:38.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:21:38.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:21:38.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-06-11T18:21:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:21:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:21:38.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:21:38.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:21:38.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:21:38.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:21:38.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:21:38.369 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:21:38.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:40.587 INFO:tasks.workunit.client.0.smithi012.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-11T18:21:41.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:21:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:21:41.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:21:41.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:21:41.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:21:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:21:41.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:21:41.134 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:21:41.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:21:41.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:21:41.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:21:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:21:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:21:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:21:41.169 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:41.164+0000 7f64e4d9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:41.172 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:41.168+0000 7f64e4d9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:41.173 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:41.168+0000 7f64e4d9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:41.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:21:41.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:21:41.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:41.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:41.709 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:41.704+0000 7f64e4d9e7c0 -1 Falling back to public interface 2024-06-11T18:21:41.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:42.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:42.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:42.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:21:42.841 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:21:42.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:42.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:43.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:44.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:44.179 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:21:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:21:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:44.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:44.605 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:44.600+0000 7f64e4d9e7c0 -1 osd.0 197 log_to_monitors true 2024-06-11T18:21:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:21:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:21:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:21:45.971 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 201 up_thru 195 down_at 198 last_clean_interval [189,197) [v2:127.0.0.1:6802/3211318618,v1:127.0.0.1:6803/3211318618] [v2:127.0.0.1:6804/3211318618,v1:127.0.0.1:6805/3211318618] exists,up 8afc0318-49ee-4091-8db9-38fea222c77d 2024-06-11T18:21:45.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:21:45.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:21:45.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:21:45.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:21:45.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:21:45.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:21:45.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:21:45.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:21:45.973 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:21:45.973 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:21:45.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:21:45.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:21:45.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:21:46.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:21:46.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:21:46.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:21:46.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:21:46.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:21:46.106 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:21:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:21:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:21:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:21:46.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:46.743 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:21:46.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=863288426498 2024-06-11T18:21:46.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 863288426498 2024-06-11T18:21:46.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426498' 2024-06-11T18:21:46.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:21:46.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:21:47.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=837518622725 2024-06-11T18:21:47.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 837518622725 2024-06-11T18:21:47.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426498 1-837518622725' 2024-06-11T18:21:47.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:47.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-863288426498 2024-06-11T18:21:47.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:47.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:21:47.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-863288426498 2024-06-11T18:21:47.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=863288426498 2024-06-11T18:21:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-06-11T18:21:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 863288426498 2024-06-11T18:21:47.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:21:47.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 863288426498 2024-06-11T18:21:47.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:21:48.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:21:48.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:21:48.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 863288426498 -lt 863288426498 2024-06-11T18:21:48.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:21:48.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-837518622725 2024-06-11T18:21:48.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:21:48.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:21:48.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-837518622725 2024-06-11T18:21:48.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:21:48.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=837518622725 2024-06-11T18:21:48.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 837518622725' 2024-06-11T18:21:48.842 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 837518622725 2024-06-11T18:21:48.842 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:21:49.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 837518622725 -lt 837518622725 2024-06-11T18:21:49.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:21:49.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:49.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:49.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:21:49.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:49.628 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:49.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:49.981 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:49.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:49.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:50.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:50.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:21:50.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:21:50.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:21:50.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:50.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:21:50.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:50.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:50.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:50.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:21:50.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:50.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:50.796 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:51.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:21:51.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:21:51.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:21:51.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:21:51.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:21:51.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:21:51.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:21:51.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:21:51.181 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:21:51.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:21:51.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:21:51.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:21:51.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:21:51.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:21:51.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:21:51.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:21:51.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:21:51.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:21:51.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:21:51.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:21:51.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:21:51.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:21:52.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:21:52.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:21:52.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:21:52.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:21:52.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:21:52.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:21:53.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:21:53.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:53.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:21:53.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:21:53.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:21:53.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:21:53.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:21:53.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-11T18:21:54.670 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-11T18:21:54.670 INFO:tasks.workunit.client.0.smithi012.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:21:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:21:55.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:21:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:21:55.209 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:21:55.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:21:55.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:21:55.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:21:55.211 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:21:55.211 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:21:55.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:21:55.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:21:55.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:21:55.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:21:55.214 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:21:55.249 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:55.244+0000 7f2254fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:55.252 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:55.248+0000 7f2254fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:55.253 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:55.248+0000 7f2254fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:21:55.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:55.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:55.800 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:55.796+0000 7f2254fca7c0 -1 Falling back to public interface 2024-06-11T18:21:55.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:56.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:21:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:21:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:56.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:57.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:58.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:58.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:58.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:21:58.231 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:21:58.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:58.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:21:58.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:21:58.400+0000 7f2254fca7c0 -1 osd.1 203 log_to_monitors true 2024-06-11T18:21:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:21:59.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:21:59.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:21:59.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:21:59.648 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:21:59.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:21:59.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:22:00.283 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 207 up_thru 207 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/2078411731,v1:127.0.0.1:6811/2078411731] [v2:127.0.0.1:6812/2078411731,v1:127.0.0.1:6813/2078411731] exists,up b2541df2-1a8b-47b0-9ae2-f9de633bebc8 2024-06-11T18:22:00.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:22:00.284 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:22:00.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:22:00.285 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:22:00.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:22:00.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:22:00.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:22:00.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:22:00.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:22:00.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:22:00.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:22:00.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:22:00.415 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:22:00.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:22:00.853 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:22:00.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:22:00.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:22:00.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:22:01.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=863288426502 2024-06-11T18:22:01.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 863288426502 2024-06-11T18:22:01.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426502' 2024-06-11T18:22:01.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:22:01.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:22:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=889058230274 2024-06-11T18:22:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 889058230274 2024-06-11T18:22:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426502 1-889058230274' 2024-06-11T18:22:01.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:22:01.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:22:01.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-863288426502 2024-06-11T18:22:01.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:22:01.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-863288426502 2024-06-11T18:22:01.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:22:01.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=863288426502 2024-06-11T18:22:01.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 863288426502' 2024-06-11T18:22:01.249 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 863288426502 2024-06-11T18:22:01.250 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:22:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 863288426502 -lt 863288426502 2024-06-11T18:22:01.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:22:01.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:22:01.616 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-889058230274 2024-06-11T18:22:01.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:22:01.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-889058230274 2024-06-11T18:22:01.619 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:22:01.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=889058230274 2024-06-11T18:22:01.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-06-11T18:22:01.620 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 889058230274 2024-06-11T18:22:01.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:22:01.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 889058230274 -lt 889058230274 2024-06-11T18:22:01.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:22:01.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:22:01.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:22:02.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:22:02.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:22:02.412 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:22:02.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:22:02.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:22:02.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:22:02.741 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:22:03.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:22:03.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:22:03.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:22:03.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-06-11T18:22:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-11T18:22:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:22:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:22:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-06-11T18:22:03.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ11 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ11 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:22:03.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-11T18:22:03.130 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-11T18:22:03.130 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:22:03.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:22:03.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-11T18:22:03.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:22:03.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-11T18:22:03.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:22:03.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T18:22:03.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:22:03.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:22:03.451 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:22:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T18:22:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:22:03.452 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:22:03.453 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:03.453 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:03.453 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:03.453 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:22:03.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:22:03.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:22:03.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:22:03.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:22:03.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-06-11T18:22:03.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:22:03.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:22:03.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:03.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:03.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:03.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:22:03.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:22:03.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-11T18:22:03.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=mdata 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ12 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-06-11T18:22:03.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:22:03.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ12 2024-06-11T18:22:03.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:22:03.706 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-11T18:22:03.707 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-11T18:22:03.707 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:22:04.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-11T18:22:04.028 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:22:04.029 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:22:04.030 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:04.030 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:04.030 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:04.030 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:22:04.031 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:22:04.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:22:04.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:22:04.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:22:04.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-06-11T18:22:04.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:22:04.145 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:22:04.146 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:04.146 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:04.146 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:04.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:22:04.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:22:04.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-11T18:22:04.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=mdata 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ13 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:22:04.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ13 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-11T18:22:04.277 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:22:04.600 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:22:04.600 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:22:04.600 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:22:04.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-11T18:22:04.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:22:04.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-11T18:22:04.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:22:04.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-11T18:22:04.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:22:04.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:22:04.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:22:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-11T18:22:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:22:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:22:04.604 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:22:04.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:22:04.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:22:04.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:22:04.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:22:04.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-06-11T18:22:04.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-11T18:22:04.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-11T18:22:04.722 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:04.722 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:04.722 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:04.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.0.asok 2024-06-11T18:22:04.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:22:04.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=csr_pool 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=ROBJ13 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-scrub-repair 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-06-11T18:22:04.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-06-11T18:22:04.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-06-11T18:22:04.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds csr_pool ROBJ13 2024-06-11T18:22:04.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-11T18:22:04.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-11T18:22:04.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-11T18:22:04.837 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:22:05.154 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-11T18:22:05.154 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:22:05.154 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-11T18:22:05.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0') 2024-06-11T18:22:05.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-06-11T18:22:05.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-06-11T18:22:05.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-06-11T18:22:05.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-06-11T18:22:05.156 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-11T18:22:05.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-06-11T18:22:05.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-11T18:22:05.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-11T18:22:05.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-11T18:22:05.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-11T18:22:05.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:05.158 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:05.159 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:22:05.159 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61904/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-11T18:22:05.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-11T18:22:05.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-06-11T18:22:05.280 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-06-11T18:22:05.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-06-11T18:22:05.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-11T18:22:05.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-11T18:22:05.281 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:05.281 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:05.281 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:05.281 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-osd.1.asok 2024-06-11T18:22:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-06-11T18:22:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61904/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-11T18:22:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-06-11T18:22:05.526 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0: {} 2024-06-11T18:22:05.526 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0: osd_max_object_size = '' 2024-06-11T18:22:05.541 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1: {} 2024-06-11T18:22:05.542 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1: osd_max_object_size = '' 2024-06-11T18:22:05.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-11T18:22:05.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: local pgid=3.0 2024-06-11T18:22:05.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:22:05.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:05.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:22:05.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:05.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:22:05.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: local last_scrub=2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:05.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-11T18:22:06.186 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: pg_deep_scrub: wait_for_scrub 3.0 2024-06-11T18:16:10.313337+0000 last_deep_scrub_stamp 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:22:06.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:06.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:22:06.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:06.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:22:06.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:06.205 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:22:06.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:06.539 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:07.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:07.541 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:07.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:22:07.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:07.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:22:07.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:07.543 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:22:07.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:07.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:08.881 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:08.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:08.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:22:08.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:08.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:22:08.883 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:08.884 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:22:09.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:16:10.313337+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:09.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:10.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:10.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:10.220 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-11T18:22:10.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:10.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:22:10.221 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:10.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-11T18:22:10.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:22:06.749590+0000 '>' 2024-06-11T18:16:10.313337+0000 2024-06-11T18:22:10.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:22:10.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-06-11T18:22:10.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-11T18:22:10.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-06-11T18:22:10.558 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.598 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.666 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.859 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:10.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:10.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:11.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:11.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:11.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-11T18:22:11.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-06-11T18:22:11.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-11T18:22:11.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:22:11.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-11T18:22:11.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:22:11.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-11T18:22:11.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-11T18:22:11.162 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-11T18:22:11.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:22:11.193 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-11T18:22:11.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:22:11.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:22:11.248 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:22:11.249 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:22:11.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-11T18:22:11.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:22:11.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:22:11.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:22:11.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-06-11T18:22:11.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-11T18:22:11.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-06-11T18:22:11.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=3.0 2024-06-11T18:22:11.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 3.0 2024-06-11T18:22:11.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:11.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:22:11.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:11.304 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:22:11.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:11.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 3.0 2024-06-11T18:22:11.945 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0 on osd.1 to repair 2024-06-11T18:22:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 3.0 2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:22:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:22:11.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:22:11.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:11.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:22:11.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:11.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:22:11.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:11.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:22:12.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:22:06.749590+0000 '>' 2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:12.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:13.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:13.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:13.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:22:13.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:13.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:22:13.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:13.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:22:13.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:22:06.749590+0000 '>' 2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:13.637 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:14.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:14.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:14.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:22:14.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:14.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:22:14.640 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:14.641 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:22:14.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:22:06.749590+0000 '>' 2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:14.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:22:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:22:15.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:22:15.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:22:15.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:22:15.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:22:15.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:22:15.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:22:16.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:22:13.719452+0000 '>' 2024-06-11T18:22:06.749590+0000 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:22:16.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:22:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:22:16.313 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:22:16.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:22:16.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:22:16.313 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:22:16.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:22:16.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:22:16.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:22:16.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:22:16.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:22:16.431 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:22:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:22:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:22:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:22:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:22:16.776 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:22:16.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=863288426506 2024-06-11T18:22:16.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 863288426506 2024-06-11T18:22:16.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426506' 2024-06-11T18:22:16.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:22:16.930 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:22:17.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=889058230278 2024-06-11T18:22:17.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 889058230278 2024-06-11T18:22:17.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-863288426506 1-889058230278' 2024-06-11T18:22:17.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:22:17.067 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-863288426506 2024-06-11T18:22:17.067 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:22:17.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:22:17.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-863288426506 2024-06-11T18:22:17.069 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:22:17.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=863288426506 2024-06-11T18:22:17.070 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 863288426506 2024-06-11T18:22:17.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 863288426506' 2024-06-11T18:22:17.071 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:22:17.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 863288426505 -lt 863288426506 2024-06-11T18:22:17.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:22:18.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:22:18.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:22:18.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 863288426506 -lt 863288426506 2024-06-11T18:22:18.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:22:18.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-889058230278 2024-06-11T18:22:18.715 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:22:18.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:22:18.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-889058230278 2024-06-11T18:22:18.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:22:18.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=889058230278 2024-06-11T18:22:18.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 889058230278' 2024-06-11T18:22:18.721 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 889058230278 2024-06-11T18:22:18.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:22:19.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 889058230278 -lt 889058230278 2024-06-11T18:22:19.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:22:19.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:22:19.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:22:19.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:22:19.544 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:22:19.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:22:19.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:22:19.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:22:19.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:22:19.545 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:22:19.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:22:19.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:22:19.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:22:19.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:22:19.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:22:20.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:22:20.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:22:20.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:22:20.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-11T18:22:20.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-11T18:22:20.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-11T18:22:20.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-11T18:22:20.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-11T18:22:20.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-11T18:22:20.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-06-11T18:22:20.481 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-11T18:22:20.936 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'csr_pool' removed 2024-06-11T18:22:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:22:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:22:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:22:20.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:22:20.961 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:22:20.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:22:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:22:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:22:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:22:21.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:22:21.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:22:21.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:22:21.077 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:22:21.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:22:21.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:22:21.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:22:21.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:22:21.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:22:21.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:22:21.080 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:22:21.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:22:21.081 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:22:21.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:22:21.123 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:22:21.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:21.124 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:21.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:22:21.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:22:21.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:22:21.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:22:21.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:22:21.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:22:21.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:22:21.128 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:22:21.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:22:21.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:22:21.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:22:21.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:22:21.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:22:21.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:22:21.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:22:21.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:22:21.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:22:21.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:22:21.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:22:21.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:22:21.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:22:21.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:21.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:21.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:22:21.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:22:21.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:22:21.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:22:21.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:22:21.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:21.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:21.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-11T18:22:21.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:22:21.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:22:21.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:22:21.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:22:21.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:22:21.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-11T18:22:21.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:22:21.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:22:21.672 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:22:21.672 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:22:21.672 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:21.673 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:21.673 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:22:21.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-11T18:22:21.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:22:21.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:22:21.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:22:21.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:22:21.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:22:21.953 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:22:21.953 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:22:21.953 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:22:21.953 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:22:21.954 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:21.954 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:21.954 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:21.954 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:22:21.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:22:21.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:22:22.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:22:22.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:22:22.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:22:22.072 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:22:22.073 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:22:22.073 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:22:22.073 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:22:22.073 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:22:22.074 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:22:22.074 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:22.074 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:22.074 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:22:22.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:22:22.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:22:22.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-06-11T18:22:22.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:22:22.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:22:22.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:22:22.186 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:22:22.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:22:22.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:22:23.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:22:23.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:22:23.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:22:23.902 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:22:23.902 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:23.902 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:23.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:22:23.903 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:22:23.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:22:23.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:22:23.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:22:23.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:22:23.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:22:23.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:22:23.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:22:23.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:22:23.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=660f8eee-a96f-415a-a741-9482e7e977df 2024-06-11T18:22:23.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 660f8eee-a96f-415a-a741-9482e7e977df' 2024-06-11T18:22:23.950 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 660f8eee-a96f-415a-a741-9482e7e977df 2024-06-11T18:22:23.951 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:22:23.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDflWhm3zyNORAAe3QKQxXosmjlwH/V1vp8ew== 2024-06-11T18:22:23.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDflWhm3zyNORAAe3QKQxXosmjlwH/V1vp8ew=="}' 2024-06-11T18:22:23.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 660f8eee-a96f-415a-a741-9482e7e977df -i td/osd-scrub-repair/0/new.json 2024-06-11T18:22:24.244 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:22:24.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:22:24.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDflWhm3zyNORAAe3QKQxXosmjlwH/V1vp8ew== --osd-uuid 660f8eee-a96f-415a-a741-9482e7e977df 2024-06-11T18:22:24.295 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.292+0000 7f10205857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:24.328 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.324+0000 7f10205857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:24.329 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.324+0000 7f10205857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:24.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.324+0000 7f10205857c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:22:24.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.324+0000 7f10205857c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:22:24.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.324+0000 7f10205857c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:22:24.436 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:24.432+0000 7f10205857c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:22:32.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:22:32.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:22:32.243 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:22:32.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:22:32.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:22:32.646 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:22:32.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:22:32.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:22:32.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:22:32.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:22:32.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:22:32.681 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:32.676+0000 7f8a8c3887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:32.725 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:32.720+0000 7f8a8c3887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:32.726 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:32.724+0000 7f8a8c3887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:22:32.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:32.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:22:32.969 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:22:32.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:32.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:33.281 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:33.276+0000 7f8a8c3887c0 -1 Falling back to public interface 2024-06-11T18:22:33.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:34.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:34.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:34.347 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:22:34.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:22:34.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:34.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:34.684 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:35.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:35.686 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:35.686 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:22:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:22:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:35.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:36.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:36.153 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:22:36.148+0000 7f8a8c3887c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:22:37.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:37.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:37.015 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:22:37.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:22:37.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:37.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:37.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:38.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:38.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:38.433 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:22:38.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:22:38.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:38.434 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:38.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:39.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:39.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:39.774 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:22:39.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:22:39.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:39.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:40.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:41.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:41.132 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:22:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:22:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:41.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:41.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:42.586 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:22:42.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:22:42.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:42.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:42.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:43.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:43.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:43.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:22:43.925 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:22:43.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:43.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:44.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:45.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:45.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:45.401 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:22:45.402 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:22:45.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:45.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:45.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:46.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:46.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:46.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:22:46.737 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:22:46.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:46.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:47.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:48.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:48.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:48.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:22:48.199 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:22:48.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:48.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:48.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:49.671 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:49.672 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:49.672 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:22:49.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:22:49.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:49.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:50.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:51.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:51.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:51.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:22:51.002 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:22:51.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:51.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:51.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:52.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:52.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:52.343 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:22:52.344 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:22:52.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:52.344 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:52.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:53.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:53.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:53.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:22:53.730 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:22:53.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:53.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:54.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:55.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:22:55.198 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:22:55.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:55.198 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:55.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:56.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:56.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:56.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:22:56.645 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:22:56.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:56.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:57.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:22:58.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:22:58.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:22:58.275 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:22:58.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:22:58.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:22:58.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:22:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:00.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:00.071 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:00.071 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:23:00.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:23:00.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:00.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:00.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:01.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:01.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:01.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:23:01.559 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:23:01.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:01.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:02.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:03.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:23:03.304 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:23:03.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:03.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:04.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:05.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:05.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:05.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:23:05.251 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:23:05.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:05.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:05.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:06.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:06.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:06.971 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:23:06.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:23:06.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:06.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:07.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:08.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:08.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:08.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:23:08.313 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:23:08.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:08.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:23:08.648 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2832479356,v1:127.0.0.1:6803/2832479356] [v2:127.0.0.1:6804/2832479356,v1:127.0.0.1:6805/2832479356] exists,up 660f8eee-a96f-415a-a741-9482e7e977df 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:23:08.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:23:08.652 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:23:08.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:23:08.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:23:08.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:23:08.655 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:23:08.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc15ade0-8e18-4a69-9070-ac89b3d77ebb 2024-06-11T18:23:08.656 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 cc15ade0-8e18-4a69-9070-ac89b3d77ebb 2024-06-11T18:23:08.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cc15ade0-8e18-4a69-9070-ac89b3d77ebb' 2024-06-11T18:23:08.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:23:08.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMlmhm5aCCKBAAMapDbQIir+VCSNH2eguD+g== 2024-06-11T18:23:08.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMlmhm5aCCKBAAMapDbQIir+VCSNH2eguD+g=="}' 2024-06-11T18:23:08.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc15ade0-8e18-4a69-9070-ac89b3d77ebb -i td/osd-scrub-repair/1/new.json 2024-06-11T18:23:09.125 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:23:09.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:23:09.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMlmhm5aCCKBAAMapDbQIir+VCSNH2eguD+g== --osd-uuid cc15ade0-8e18-4a69-9070-ac89b3d77ebb 2024-06-11T18:23:09.190 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.184+0000 7f02e3fc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:09.290 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.284+0000 7f02e3fc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.288+0000 7f02e3fc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.288+0000 7f02e3fc27c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:23:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.288+0000 7f02e3fc27c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:23:09.292 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.288+0000 7f02e3fc27c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:23:09.343 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:09.340+0000 7f02e3fc27c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:23:17.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:23:17.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:23:17.473 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:23:17.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:23:17.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:23:17.876 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:23:17.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:23:17.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:23:17.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:23:17.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:23:17.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:23:17.910 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:17.904+0000 7fe6ca8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:17.957 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:17.952+0000 7fe6ca8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:17.959 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:17.956+0000 7fe6ca8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:23:18.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:23:18.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:23:18.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:23:18.190 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:23:18.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:23:18.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:23:18.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:18.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:23:18.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:18.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:18.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:18.528 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:18.524+0000 7fe6ca8ec7c0 -1 Falling back to public interface 2024-06-11T18:23:19.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:19.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:19.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:23:19.506 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:23:19.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:19.506 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:19.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:20.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:20.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:20.843 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:23:20.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:23:20.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:20.844 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:21.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:21.587 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:23:21.564+0000 7fe6ca8ec7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:23:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:22.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:22.258 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:23:22.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:23:22.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:22.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:22.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:23.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:23.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:23.753 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:23:23.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:23:23.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:23.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:24.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:25.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:25.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:25.126 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:23:25.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:23:25.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:25.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:25.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:26.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:26.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:26.625 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:23:26.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:23:26.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:26.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:26.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:27.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:27.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:27.968 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:23:27.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:23:27.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:27.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:28.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:29.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:29.303 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:29.303 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:23:29.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:23:29.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:29.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:29.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:30.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:30.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:30.762 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:23:30.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:23:30.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:30.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:31.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:32.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:32.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:32.088 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:23:32.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:23:32.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:32.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:32.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:33.487 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:23:33.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:23:33.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:33.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:33.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:34.893 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:23:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:23:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:35.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:36.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:36.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:36.323 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:23:36.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:23:36.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:36.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:36.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:37.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:37.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:37.739 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:23:37.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:23:37.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:37.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:38.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:39.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:39.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:39.182 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:23:39.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:23:39.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:39.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:39.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:40.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:40.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:40.551 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:23:40.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:23:40.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:40.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:40.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:41.875 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:41.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:41.876 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:23:41.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:23:41.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:41.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:42.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:43.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:43.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:43.363 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:23:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:23:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:44.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:45.042 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:23:45.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:23:45.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:45.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:45.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:46.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:46.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:46.795 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:23:46.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:23:46.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:46.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:47.181 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:48.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:48.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:48.184 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:23:48.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:23:48.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:48.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:48.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:49.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:49.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:49.816 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:23:49.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:23:49.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:49.817 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:50.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:51.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:51.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:51.531 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:23:51.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:23:51.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:51.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:52.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:53.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:53.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:53.392 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:23:53.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:23:53.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:53.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:53.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:23:54.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:23:54.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:23:54.783 INFO:tasks.workunit.client.0.smithi012.stdout:25 2024-06-11T18:23:54.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-11T18:23:54.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:23:54.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:23:55.152 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2471554118,v1:127.0.0.1:6811/2471554118] [v2:127.0.0.1:6812/2471554118,v1:127.0.0.1:6813/2471554118] exists,up cc15ade0-8e18-4a69-9070-ac89b3d77ebb 2024-06-11T18:23:55.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:23:55.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:23:55.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:23:55.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-06-11T18:23:55.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:23:55.491 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:23:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:23:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:23:55.987 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:23:56.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:23:57.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:24:00.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-11T18:24:00.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:24:00.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:24:00.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:24:00.035 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:24:00.036 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:24:00.036 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:24:00.036 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:24:00.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:24:00.037 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:24:00.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:24:00.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:24:00.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:24:00.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:24:00.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:24:00.177 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:24:00.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:24:00.511 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:24:00.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:24:00.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:00.512 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:24:00.693 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-06-11T18:24:00.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-06-11T18:24:00.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-06-11T18:24:00.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:00.694 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672969' 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:00.841 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-06-11T18:24:00.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:24:00.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-06-11T18:24:00.844 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:00.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-06-11T18:24:00.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-11T18:24:00.846 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836499 2024-06-11T18:24:00.846 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:01.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-11T18:24:01.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:01.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-06-11T18:24:01.225 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:01.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:24:01.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-06-11T18:24:01.229 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:01.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-06-11T18:24:01.230 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672969 2024-06-11T18:24:01.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-11T18:24:01.231 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:24:01.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-06-11T18:24:01.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:24:01.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:01.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:24:02.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:02.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:24:02.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:02.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:02.381 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:02.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:24:02.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:24:02.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:24:02.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-06-11T18:24:02.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-11T18:24:03.273 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'foo' created 2024-06-11T18:24:03.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:24:04.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-06-11T18:24:04.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-11T18:24:04.912 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'csr_pool' created 2024-06-11T18:24:04.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:24:05.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-11T18:24:05.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:24:05.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:24:05.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:24:05.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:24:05.940 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:24:05.940 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:24:05.940 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:24:05.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:24:05.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:24:06.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:24:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:24:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:24:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:24:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:24:06.087 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:24:06.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:24:06.479 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:24:06.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:24:06.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:06.480 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:24:06.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836501 2024-06-11T18:24:06.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836501 2024-06-11T18:24:06.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501' 2024-06-11T18:24:06.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:06.634 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:24:06.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672971 2024-06-11T18:24:06.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672971 2024-06-11T18:24:06.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672971' 2024-06-11T18:24:06.782 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:06.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836501 2024-06-11T18:24:06.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:06.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:24:06.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836501 2024-06-11T18:24:06.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:06.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836501 2024-06-11T18:24:06.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-11T18:24:06.786 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836501 2024-06-11T18:24:06.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:07.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-11T18:24:07.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:07.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672971 2024-06-11T18:24:07.111 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:07.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:24:07.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672971 2024-06-11T18:24:07.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:07.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672971 2024-06-11T18:24:07.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-06-11T18:24:07.116 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672971 2024-06-11T18:24:07.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:24:07.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672971 -lt 42949672971 2024-06-11T18:24:07.436 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:24:07.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:07.437 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:07.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:24:07.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:07.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:07.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:07.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:07.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:07.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:07.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:08.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:24:08.246 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:08.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:08.247 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:08.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:24:08.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:24:08.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:24:08.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:24:08.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:24:08.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:24:09.567 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:24:09.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:24:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:24:10.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:24:10.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:24:10.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:24:10.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-11T18:24:10.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-11T18:24:10.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:24:10.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:24:11.473 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:24:11.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:24:11.974 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:24:11.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:24:11.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:24:11.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:24:12.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-11T18:24:12.656 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-11T18:24:12.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-06-11T18:24:12.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-11T18:24:12.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-11T18:24:12.824 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-11T18:24:12.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:24:13.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-06-11T18:24:13.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-06-11T18:24:13.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-11T18:24:13.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-11T18:24:13.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:24:13.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-11T18:24:13.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-06-11T18:24:13.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-06-11T18:24:13.830 INFO:tasks.workunit.client.0.smithi012.stdout:created pool csr_pool snap snap1 2024-06-11T18:24:13.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-06-11T18:24:13.835 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-11T18:24:13.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-11T18:24:13.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-11T18:24:13.837 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-06-11T18:24:13.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-06-11T18:24:13.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-11T18:24:13.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-11T18:24:14.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:24:14.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:24:14.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-11T18:24:14.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:24:14.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:24:14.154 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:24:14.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:24:14.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:24:14.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:24:14.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:24:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:24:14.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:24:17.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:24:17.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:24:17.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:24:17.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:24:17.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:24:17.167 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:24:17.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:24:17.167 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:24:17.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:24:17.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:24:17.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:24:17.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:24:17.202 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:17.195+0000 7f83833cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:17.205 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:17.199+0000 7f83833cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:17.207 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:17.203+0000 7f83833cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:17.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:17.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:24:17.740 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:17.735+0000 7f83833cf7c0 -1 Falling back to public interface 2024-06-11T18:24:17.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:18.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:18.888 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:18.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:24:18.889 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:24:18.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:18.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:24:19.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:20.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:20.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:20.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:24:20.231 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:24:20.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:20.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:24:20.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:20.583 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:20.579+0000 7f83833cf7c0 -1 osd.1 31 log_to_monitors true 2024-06-11T18:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:21.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:21.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:24:22.159 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 35 up_thru 23 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/389395966,v1:127.0.0.1:6811/389395966] [v2:127.0.0.1:6812/389395966,v1:127.0.0.1:6813/389395966] exists,up cc15ade0-8e18-4a69-9070-ac89b3d77ebb 2024-06-11T18:24:22.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:24:22.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:24:22.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:24:22.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:24:22.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:24:22.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:24:22.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:24:22.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:24:22.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:24:22.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:24:22.788 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:24:22.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:24:22.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:22.789 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:24:23.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-06-11T18:24:23.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-06-11T18:24:23.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-06-11T18:24:23.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:23.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:24:23.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855362 2024-06-11T18:24:23.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855362 2024-06-11T18:24:23.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-150323855362' 2024-06-11T18:24:23.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:23.307 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-06-11T18:24:23.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:23.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:24:23.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-06-11T18:24:23.310 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:23.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-06-11T18:24:23.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-11T18:24:23.312 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 21474836506 2024-06-11T18:24:23.312 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:23.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836506 2024-06-11T18:24:23.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:24:24.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:24:24.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:25.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-11T18:24:25.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:25.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855362 2024-06-11T18:24:25.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:25.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:24:25.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855362 2024-06-11T18:24:25.144 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:25.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855362 2024-06-11T18:24:25.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-11T18:24:25.146 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 150323855362 2024-06-11T18:24:25.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:24:25.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-11T18:24:25.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:24:25.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:25.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:25.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:24:25.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:25.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:25.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:25.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:25.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:25.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:25.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:26.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:24:26.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:26.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:26.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:24:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-11T18:24:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-11T18:24:26.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-11T18:24:26.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:24:26.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-11T18:24:26.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:24:26.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:26.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:26.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:26.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:26.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:26.799 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:26.800 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:27.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:24:27.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:27.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:27.139 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:27.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:24:27.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:24:27.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:24:27.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:24:27.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:24:27.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:24:27.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:24:27.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:24:27.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:24:27.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:24:27.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:24:27.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-11T18:24:27.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:24:27.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-11T18:24:28.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:24:28.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:28.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:28.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:28.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:28.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:28.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:28.110 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:28.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-11T18:24:28.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:28.433 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:28.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-11T18:24:28.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-11T18:24:28.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-11T18:24:28.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-11T18:24:29.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-11T18:24:29.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-11T18:24:29.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-11T18:24:29.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-11T18:24:29.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-11T18:24:29.879 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-11T18:24:29.880 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:29.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:29.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:29.880 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:29.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:29.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:29.881 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:30.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:24:30.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:30.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:30.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:30.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:24:30.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:24:30.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:24:30.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-11T18:24:30.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-11T18:24:30.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-06-11T18:24:30.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-06-11T18:24:30.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-11T18:24:30.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:24:30.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:24:30.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:24:30.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:24:30.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:24:30.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:24:30.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:24:30.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:24:30.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:24:31.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-06-11T18:24:33.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-11T18:24:33.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:24:33.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:24:33.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:33.803 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:24:33.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:24:33.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:24:33.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:24:33.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:24:33.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:24:33.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:24:33.806 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:24:33.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:24:33.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:24:33.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:24:33.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:24:33.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:24:33.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:24:33.841 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:33.835+0000 7fd0c789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:33.843 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:33.839+0000 7fd0c789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:33.845 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:33.839+0000 7fd0c789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:34.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:24:34.389 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:34.383+0000 7fd0c789e7c0 -1 Falling back to public interface 2024-06-11T18:24:34.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:24:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:24:35.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:35.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:24:35.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:24:36.838 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:24:36.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:36.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:24:37.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:24:37.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:37.343+0000 7fd0c789e7c0 -1 osd.0 37 log_to_monitors true 2024-06-11T18:24:38.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:24:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:24:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:24:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:24:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:24:38.702 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/2817082247,v1:127.0.0.1:6803/2817082247] [v2:127.0.0.1:6804/2817082247,v1:127.0.0.1:6805/2817082247] exists,up 660f8eee-a96f-415a-a741-9482e7e977df 2024-06-11T18:24:38.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:24:38.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:24:38.703 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:24:38.704 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:24:38.704 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:24:38.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:24:38.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:24:38.704 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:24:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:24:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:24:38.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:24:38.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:24:38.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:24:38.834 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:24:39.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:24:39.164 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:24:39.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:24:39.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:39.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:24:39.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=176093659138 2024-06-11T18:24:39.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 176093659138 2024-06-11T18:24:39.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-176093659138' 2024-06-11T18:24:39.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:24:39.427 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:24:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855366 2024-06-11T18:24:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855366 2024-06-11T18:24:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-176093659138 1-150323855366' 2024-06-11T18:24:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:39.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-176093659138 2024-06-11T18:24:39.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:24:39.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-176093659138 2024-06-11T18:24:39.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:39.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=176093659138 2024-06-11T18:24:39.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-06-11T18:24:39.613 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 176093659138 2024-06-11T18:24:39.613 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:39.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 176093659138 2024-06-11T18:24:39.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:24:40.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:24:40.938 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:24:41.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659138 -lt 176093659138 2024-06-11T18:24:41.317 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:24:41.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-150323855366 2024-06-11T18:24:41.318 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:24:41.320 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:24:41.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-150323855366 2024-06-11T18:24:41.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:24:41.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855366 2024-06-11T18:24:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 150323855366' 2024-06-11T18:24:41.323 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 150323855366 2024-06-11T18:24:41.323 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:24:41.691 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855366 -lt 150323855366 2024-06-11T18:24:41.691 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:24:41.691 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:41.692 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:42.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-11T18:24:42.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:24:42.141 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:24:42.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-11T18:24:42.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:24:42.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:24:42.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:24:42.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-11T18:24:42.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:24:42.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:24:42.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-06-11T18:24:42.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-06-11T18:24:42.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=3.0 2024-06-11T18:24:42.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 3.0 2024-06-11T18:24:42.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:24:42.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:24:42.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:24:42.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:24:43.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:43.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 3.0 2024-06-11T18:24:43.586 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 3.0 2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=3.0 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:24:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:24:43.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:24:43.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:24:43.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:24:43.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:24:43.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:24:43.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:24:04.784467+0000 '>' 2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:43.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:24:44.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:24:44.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:24:44.922 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:24:44.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:24:44.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:24:44.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:24:44.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:24:45.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:24:04.784467+0000 '>' 2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:45.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:24:46.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:24:46.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:24:46.257 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:24:46.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:24:46.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:24:46.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:24:46.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:24:46.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:24:04.784467+0000 '>' 2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:46.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:24:47.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:24:47.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:24:47.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-11T18:24:47.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-11T18:24:47.590 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:24:47.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-11T18:24:47.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:24:47.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:24:44.606074+0000 '>' 2024-06-11T18:24:04.784467+0000 2024-06-11T18:24:47.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:24:47.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-06-11T18:24:47.957 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-06-11T18:24:47.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-06-11T18:24:47.986 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-11T18:24:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-06-11T18:24:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:24:48.059 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:24:48.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-11T18:24:48.060 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-11T18:24:48.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: . as $in 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: | if type == "object" then 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: reduce keys[] as $key 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: else f 2024-06-11T18:24:48.096 INFO:tasks.workunit.client.0.smithi012.stderr: end; 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-11T18:24:48.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-11T18:24:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:24:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-11T18:24:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-06-11T18:24:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-06-11T18:24:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-11T18:24:48.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-06-11T18:24:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-06-11T18:24:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-11T18:24:48.593 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'csr_pool' removed 2024-06-11T18:24:48.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:24:48.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:24:48.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:24:48.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:24:48.617 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:24:48.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:24:48.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:24:48.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:24:48.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:24:48.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:24:48.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:24:48.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:24:48.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:24:48.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:24:48.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:24:48.939 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:24:48.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:24:48.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:24:48.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:24:48.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:24:48.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:24:48.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:24:48.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:24:48.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:24:48.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:48.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:48.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:24:48.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:24:48.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:24:48.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:24:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:24:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:24:48.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:24:48.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:24:48.973 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:24:48.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:24:48.975 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:24:48.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:24:48.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:24:48.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:24:48.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:24:48.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:24:48.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:24:48.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:24:48.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:24:48.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:24:48.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:24:48.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:24:48.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:48.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:48.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:24:48.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:24:48.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:24:48.985 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:24:48.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:24:48.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:48.987 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:48.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:24:48.989 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:24:48.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:24:48.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:24:48.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:24:48.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:24:49.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:24:49.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:24:49.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:24:49.523 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:24:49.523 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:49.523 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:49.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:49.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:24:49.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:24:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:24:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:24:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:24:49.766 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:24:49.767 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:24:49.767 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:24:49.767 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:24:49.768 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:24:49.769 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:24:49.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:24:49.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:24:49.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:24:49.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:24:49.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:24:49.889 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:24:49.890 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:24:49.890 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:24:49.890 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:24:49.891 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:49.891 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:49.891 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:24:49.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:24:49.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:24:50.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:24:50.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:24:50.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:24:50.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:24:50.912 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:24:50.912 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:50.912 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:50.912 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:50.913 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:24:50.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:24:50.952 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-11T18:24:50.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:24:50.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-11T18:24:50.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:24:50.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:24:50.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:24:50.955 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:24:50.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:24:50.956 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:24:50.956 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:24:50.956 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:24:50.956 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:50.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:24:50.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:24:50.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:24:50.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4ecf5f6-38ec-4915-9ec8-701471256e4a 2024-06-11T18:24:50.959 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 b4ecf5f6-38ec-4915-9ec8-701471256e4a 2024-06-11T18:24:50.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b4ecf5f6-38ec-4915-9ec8-701471256e4a' 2024-06-11T18:24:50.959 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:24:50.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBylmhmeKIqOhAAOoG6sUQaaH3VoUH0c02F/g== 2024-06-11T18:24:50.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBylmhmeKIqOhAAOoG6sUQaaH3VoUH0c02F/g=="}' 2024-06-11T18:24:50.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4ecf5f6-38ec-4915-9ec8-701471256e4a -i td/osd-scrub-repair/0/new.json 2024-06-11T18:24:51.261 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:24:51.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:24:51.283 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBylmhmeKIqOhAAOoG6sUQaaH3VoUH0c02F/g== --osd-uuid b4ecf5f6-38ec-4915-9ec8-701471256e4a 2024-06-11T18:24:51.319 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.315+0000 7facd51ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:51.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.339+0000 7facd51ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:51.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.343+0000 7facd51ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:51.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.343+0000 7facd51ec7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:24:51.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.343+0000 7facd51ec7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:24:51.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.343+0000 7facd51ec7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:24:51.454 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:51.447+0000 7facd51ec7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:24:59.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:24:59.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:24:59.281 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:24:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:24:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:24:59.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:24:59.689 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:24:59.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:24:59.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:24:59.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:24:59.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:24:59.722 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:59.715+0000 7f3f0cd7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:59.752 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:59.747+0000 7f3f0cd7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:59.754 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:24:59.747+0000 7f3f0cd7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:24:59.992 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:24:59.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:24:59.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:24:59.993 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:00.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:00.311+0000 7f3f0cd7a7c0 -1 Falling back to public interface 2024-06-11T18:25:00.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:01.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:25:01.367 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:25:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:01.368 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:01.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:02.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:02.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:02.703 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:25:02.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:25:02.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:02.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:03.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:03.248 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:03.243+0000 7f3f0cd7a7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:25:04.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:04.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:04.039 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:25:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:25:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:04.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:04.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:05.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:05.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:05.375 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:25:05.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:25:05.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:05.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:05.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:06.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:06.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:06.708 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:25:06.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:25:06.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:06.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:07.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:08.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:08.142 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:08.142 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:25:08.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:25:08.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:08.143 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:08.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:09.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:09.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:09.604 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:25:09.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:25:09.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:09.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:09.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:10.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:10.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:10.937 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:25:10.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:25:10.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:10.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:11.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:12.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:12.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:12.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:25:12.387 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:25:12.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:12.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:12.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:13.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:13.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:13.724 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:25:13.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:25:13.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:13.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:14.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:15.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:15.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:15.175 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:25:15.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:25:15.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:15.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:15.548 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:16.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:16.550 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:16.550 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:25:16.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:25:16.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:16.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:16.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:17.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:17.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:17.889 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:25:17.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:25:17.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:17.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:18.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:19.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:19.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:19.260 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:25:19.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:25:19.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:19.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:19.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:20.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:20.626 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:25:20.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:25:20.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:20.627 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:20.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:21.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:21.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:21.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:25:21.964 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:25:21.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:21.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:22.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:23.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:23.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:23.571 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:25:23.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:25:23.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:23.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:24.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:25.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:25.190 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:25.190 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:25:25.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:25:25.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:25.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:25.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:26.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:26.808 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:26.809 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:25:26.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:25:26.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:26.809 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:27.731 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:28.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:28.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:28.734 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:25:28.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:25:28.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:28.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:29.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:30.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:30.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:30.235 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:25:30.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:25:30.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:30.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:30.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:31.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:31.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:31.847 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:25:31.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:25:31.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:31.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:32.638 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:33.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:33.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:33.640 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:25:33.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:25:33.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:33.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:34.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:35.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:35.125 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:25:35.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:25:35.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:35.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:25:35.459 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4162413842,v1:127.0.0.1:6803/4162413842] [v2:127.0.0.1:6804/4162413842,v1:127.0.0.1:6805/4162413842] exists,up b4ecf5f6-38ec-4915-9ec8-701471256e4a 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:25:35.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:25:35.464 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:25:35.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:25:35.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:25:35.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:25:35.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:25:35.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d538a17-0d08-4f0d-811d-920ed4ce207c 2024-06-11T18:25:35.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0d538a17-0d08-4f0d-811d-920ed4ce207c' 2024-06-11T18:25:35.467 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 0d538a17-0d08-4f0d-811d-920ed4ce207c 2024-06-11T18:25:35.467 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:25:35.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCflmhm5MQNHRAAoIL99HG6EXsHFqTn87P6Kw== 2024-06-11T18:25:35.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCflmhm5MQNHRAAoIL99HG6EXsHFqTn87P6Kw=="}' 2024-06-11T18:25:35.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d538a17-0d08-4f0d-811d-920ed4ce207c -i td/osd-scrub-repair/1/new.json 2024-06-11T18:25:36.014 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:25:36.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:25:36.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCflmhm5MQNHRAAoIL99HG6EXsHFqTn87P6Kw== --osd-uuid 0d538a17-0d08-4f0d-811d-920ed4ce207c 2024-06-11T18:25:36.069 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.063+0000 7fd2268057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:36.150 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.147+0000 7fd2268057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:36.151 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.147+0000 7fd2268057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:36.152 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.147+0000 7fd2268057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:25:36.152 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.147+0000 7fd2268057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:25:36.152 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.147+0000 7fd2268057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:25:36.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:36.227+0000 7fd2268057c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:25:44.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:25:44.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:25:44.163 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:25:44.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:25:44.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:25:44.576 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:25:44.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:25:44.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:25:44.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:25:44.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:25:44.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:25:44.611 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:44.606+0000 7fbe857cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:44.655 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:44.650+0000 7fbe857cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:44.656 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:44.650+0000 7fbe857cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:25:44.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:25:44.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:25:44.900 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:25:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:25:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:25:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:25:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:25:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:45.220 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:45.214+0000 7fbe857cd7c0 -1 Falling back to public interface 2024-06-11T18:25:45.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:46.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:46.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:46.278 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:25:46.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:25:46.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:46.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:46.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:47.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:47.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:47.607 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:25:47.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:25:47.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:47.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:48.234 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:25:48.226+0000 7fbe857cd7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:25:48.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:48.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:48.940 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:25:48.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:25:48.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:48.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:49.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:50.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:50.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:50.322 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:25:50.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:25:50.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:50.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:50.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:51.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:51.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:51.767 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:25:51.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:25:51.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:51.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:52.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:53.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:53.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:53.109 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:25:53.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:25:53.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:53.109 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:53.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:54.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:54.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:54.512 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:25:54.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:25:54.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:54.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:54.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:55.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:55.850 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:25:55.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:25:55.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:55.851 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:57.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:57.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:57.271 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:25:57.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:25:57.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:57.271 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:25:58.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:25:58.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:25:58.753 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:25:58.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:25:58.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:25:58.754 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:25:59.123 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:00.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:00.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:00.125 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:26:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:26:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:00.126 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:00.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:01.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:01.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:01.565 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:26:01.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:26:01.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:01.566 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:01.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:02.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:02.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:02.899 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:26:02.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:26:02.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:02.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:03.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:04.240 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:26:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:26:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:05.580 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:05.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:05.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:26:05.581 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:26:05.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:05.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:05.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:06.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:06.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:06.918 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:26:06.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:26:06.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:06.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:07.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:08.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:08.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:08.260 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:26:08.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:26:08.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:08.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:09.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:10.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:10.118 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:26:10.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:26:10.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:10.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:10.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:11.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:11.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:11.954 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:26:11.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:26:11.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:11.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:12.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:13.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:13.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:13.959 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:26:13.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:26:13.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:13.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:15.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:16.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:16.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:16.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:26:16.047 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:26:16.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:16.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:16.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:17.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:17.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:17.496 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:26:17.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:26:17.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:17.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:17.858 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:18.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:18.861 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:18.862 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:26:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:26:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:18.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:19.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:20.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:20.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:20.516 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:26:20.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:26:20.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:20.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:26:20.942 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3416675850,v1:127.0.0.1:6811/3416675850] [v2:127.0.0.1:6812/3416675850,v1:127.0.0.1:6813/3416675850] exists,up 0d538a17-0d08-4f0d-811d-920ed4ce207c 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:26:20.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:26:20.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:26:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:26:20.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:26:20.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=816a2b1b-37a4-4ef9-9f9b-2722aa607e94 2024-06-11T18:26:20.950 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 816a2b1b-37a4-4ef9-9f9b-2722aa607e94 2024-06-11T18:26:20.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 816a2b1b-37a4-4ef9-9f9b-2722aa607e94' 2024-06-11T18:26:20.950 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:26:20.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMlmhmxkGgORAAprJfbntUPkHLMrVl3UA9vw== 2024-06-11T18:26:20.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMlmhmxkGgORAAprJfbntUPkHLMrVl3UA9vw=="}' 2024-06-11T18:26:20.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 816a2b1b-37a4-4ef9-9f9b-2722aa607e94 -i td/osd-scrub-repair/2/new.json 2024-06-11T18:26:21.465 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:26:21.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T18:26:21.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMlmhmxkGgORAAprJfbntUPkHLMrVl3UA9vw== --osd-uuid 816a2b1b-37a4-4ef9-9f9b-2722aa607e94 2024-06-11T18:26:21.526 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.522+0000 7f0b3ccb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:21.600 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.598+0000 7f0b3ccb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:21.602 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.602+0000 7f0b3ccb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:21.603 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.602+0000 7f0b3ccb17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:26:21.603 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.602+0000 7f0b3ccb17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:26:21.603 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.602+0000 7f0b3ccb17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:26:21.687 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:21.686+0000 7f0b3ccb17c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T18:26:29.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T18:26:29.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:26:29.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T18:26:29.795 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T18:26:29.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:26:30.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T18:26:30.227 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T18:26:30.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:26:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:26:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:26:30.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:26:30.260 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:30.258+0000 7f29fb6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:30.290 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:30.286+0000 7f29fb6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:30.291 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:30.290+0000 7f29fb6a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:30.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:26:30.575 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:26:30.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:30.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:30.877 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:30.874+0000 7f29fb6a27c0 -1 Falling back to public interface 2024-06-11T18:26:30.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:31.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:31.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:31.964 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:26:31.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:26:31.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:31.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:32.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:33.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:33.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:33.291 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:26:33.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:26:33.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:33.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:33.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:33.785 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:26:33.782+0000 7f29fb6a27c0 -1 osd.2 0 log_to_monitors true 2024-06-11T18:26:34.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:34.623 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:34.623 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:26:34.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:26:34.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:34.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:34.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:35.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:35.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:35.970 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:26:35.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:26:35.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:35.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:36.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:37.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:37.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:37.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:26:37.415 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:26:37.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:37.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:37.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:38.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:38.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:38.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:26:38.827 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:26:38.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:38.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:39.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:40.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:40.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:40.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:26:40.522 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:26:40.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:40.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:41.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:41.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:41.932 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:26:41.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:26:41.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:41.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:42.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:43.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:43.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:43.418 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:26:43.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:26:43.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:43.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:43.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:44.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:26:44.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:26:44.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:44.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:45.310 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:46.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:46.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:46.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:26:46.313 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:26:46.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:46.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:46.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:47.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:47.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:47.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:26:47.651 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:26:47.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:47.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:48.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:49.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:49.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:49.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:26:49.233 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:26:49.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:49.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:49.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:50.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:50.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:50.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:26:50.572 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:26:50.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:50.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:50.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:51.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:51.908 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:51.908 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:26:51.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:26:51.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:51.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:52.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:53.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:53.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:53.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:26:53.452 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:26:53.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:53.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:54.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:55.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:55.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:55.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:26:55.098 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:26:55.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:55.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:55.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:56.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:56.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:56.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:26:56.705 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:26:56.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:56.705 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:57.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:26:58.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:26:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:26:58.391 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:26:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:26:58.391 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:26:58.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:26:59.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:00.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:00.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:00.188 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:27:00.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:27:00.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:00.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:00.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:01.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:27:01.963 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:27:01.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:01.964 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:02.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:03.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:03.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:03.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:27:03.740 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:27:03.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:03.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:04.834 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:05.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:05.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:05.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:27:05.836 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:27:05.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:05.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:06.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:07.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:07.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:07.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:27:07.174 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:27:07.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:07.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:07.511 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1606483360,v1:127.0.0.1:6819/1606483360] [v2:127.0.0.1:6820/1606483360,v1:127.0.0.1:6821/1606483360] exists,up 816a2b1b-37a4-4ef9-9f9b-2722aa607e94 2024-06-11T18:27:07.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:27:07.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:27:07.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:27:07.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-11T18:27:07.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:27:07.874 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:27:07.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:27:07.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:27:08.424 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:27:08.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:27:09.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:27:13.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-11T18:27:13.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:27:13.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:27:13.562 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:27:13.562 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:27:13.563 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:27:13.563 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:27:13.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:27:13.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:27:13.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:27:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:27:13.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:27:13.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:27:13.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:27:13.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:27:13.710 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:14.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:27:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-06-11T18:27:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-06-11T18:27:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-06-11T18:27:14.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:14.216 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:27:14.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-06-11T18:27:14.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-06-11T18:27:14.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-06-11T18:27:14.370 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:14.371 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:27:14.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-06-11T18:27:14.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-06-11T18:27:14.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509446' 2024-06-11T18:27:14.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:14.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-06-11T18:27:14.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:14.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:27:14.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-06-11T18:27:14.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:14.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-06-11T18:27:14.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-06-11T18:27:14.521 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836505 2024-06-11T18:27:14.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:27:14.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836505 2024-06-11T18:27:14.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:27:15.847 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:27:15.848 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:27:16.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836505 2024-06-11T18:27:16.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:16.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-06-11T18:27:16.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:16.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:27:16.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-06-11T18:27:16.262 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-06-11T18:27:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-11T18:27:16.265 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672977 2024-06-11T18:27:16.265 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:27:16.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-06-11T18:27:16.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:16.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-06-11T18:27:16.677 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:16.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:27:16.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-06-11T18:27:16.680 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-06-11T18:27:16.682 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-06-11T18:27:16.682 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509446 2024-06-11T18:27:16.682 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:27:17.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509446 2024-06-11T18:27:17.014 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:27:17.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:27:17.015 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:27:17.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:27:17.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:27:17.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:27:17.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:27:17.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:27:17.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:27:17.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:27:17.411 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:27:17.729 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:27:17.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:27:17.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:27:17.730 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T18:27:18.114 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-11T18:27:18.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T18:27:18.616 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T18:27:19.474 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T18:27:19.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:27:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T18:27:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T18:27:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:27:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:27:20.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:27:20.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:27:20.501 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:27:20.501 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:27:20.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:27:20.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:27:20.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:27:20.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:27:20.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:27:20.982 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:20.982 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:27:20.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:27:20.983 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:20.983 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:27:21.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-06-11T18:27:21.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-06-11T18:27:21.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-06-11T18:27:21.140 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:21.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:27:21.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672980 2024-06-11T18:27:21.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672980 2024-06-11T18:27:21.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-06-11T18:27:21.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:27:21.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:27:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-06-11T18:27:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-06-11T18:27:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509448' 2024-06-11T18:27:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:21.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-06-11T18:27:21.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:21.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:27:21.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-06-11T18:27:21.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:21.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-06-11T18:27:21.538 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836508 2024-06-11T18:27:21.538 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-11T18:27:21.538 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:27:21.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-06-11T18:27:21.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:27:22.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:27:22.868 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:27:23.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-11T18:27:23.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:23.202 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672980 2024-06-11T18:27:23.203 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:23.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:27:23.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-06-11T18:27:23.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:23.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672980 2024-06-11T18:27:23.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-11T18:27:23.207 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672980 2024-06-11T18:27:23.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:27:23.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-11T18:27:23.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:27:23.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-06-11T18:27:23.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:27:23.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:27:23.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-06-11T18:27:23.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:27:23.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-06-11T18:27:23.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-11T18:27:23.578 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509448 2024-06-11T18:27:23.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:27:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-11T18:27:24.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:27:24.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:27:24.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:27:24.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:27:24.419 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:27:24.420 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:27:24.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:27:24.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:27:24.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:27:24.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:27:24.421 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:27:24.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:27:24.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:27:24.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:27:24.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:27:25.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:27:25.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:27:25.663 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:27:25.685 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:27:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:27:26.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:27:26.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:27:26.272 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:27:26.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-11T18:27:26.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T18:27:26.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-11T18:27:26.657 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-11T18:27:26.657 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:27:26.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-11T18:27:26.993 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:26.993 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:27:26.993 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-11T18:27:26.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:27:26.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:27:26.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:27:27.485 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:27:27.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:27:28.037 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:27:28.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:27:28.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:27:28.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:27:28.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-11T18:27:28.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T18:27:28.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-11T18:27:28.553 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-11T18:27:28.554 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:27:28.945 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-11T18:27:28.945 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:28.945 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:27:28.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-11T18:27:28.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-11T18:27:28.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-11T18:27:28.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-11T18:27:28.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:28.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:27:28.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:27:28.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:27:28.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:27:28.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:27:28.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:27:30.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:27:30.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:30.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:27:30.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:27:30.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:27:30.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:27:30.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:27:30.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:27:31.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:27:31.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:31.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:27:31.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:27:31.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:27:31.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:27:31.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:27:31.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:27:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:27:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-11T18:27:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-11T18:27:33.558 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-11T18:27:34.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-11T18:27:34.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-11T18:27:35.736 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-11T18:27:36.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-11T18:27:36.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-11T18:27:37.332 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-11T18:27:37.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-11T18:27:37.867 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-11T18:27:39.469 INFO:tasks.workunit.client.0.smithi012.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-11T18:27:40.005 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:27:40.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:27:40.009 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:27:40.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:27:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:27:40.013 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:27:40.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:27:40.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:27:40.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:27:40.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:27:40.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:27:40.014 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:27:40.047 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:40.046+0000 7eff9c9847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:40.050 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:40.050+0000 7eff9c9847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:40.052 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:40.050+0000 7eff9c9847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:40.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:27:40.392 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:27:40.605 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:40.602+0000 7eff9c9847c0 -1 Falling back to public interface 2024-06-11T18:27:40.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:41.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:41.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:41.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:27:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:27:42.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:43.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:43.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:43.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:27:43.121 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:27:43.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:43.122 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:27:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:43.418+0000 7eff9c9847c0 -1 osd.0 35 log_to_monitors true 2024-06-11T18:27:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:44.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:44.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:44.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:27:44.457 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:27:44.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:44.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/1901143510,v1:127.0.0.1:6803/1901143510] [v2:127.0.0.1:6804/1901143510,v1:127.0.0.1:6805/1901143510] exists,up b4ecf5f6-38ec-4915-9ec8-701471256e4a 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:27:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:27:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:27:44.820 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:27:44.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:27:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:27:44.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:27:44.823 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:27:44.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:27:44.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:27:44.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:27:44.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:27:44.862 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:44.858+0000 7faea4bda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:44.895 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:44.894+0000 7faea4bda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:44.896 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:44.894+0000 7faea4bda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:27:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:27:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:27:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:45.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:27:45.428 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:45.426+0000 7faea4bda7c0 -1 Falling back to public interface 2024-06-11T18:27:45.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:46.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:46.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:46.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:27:46.514 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:46.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:46.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:27:46.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:47.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:47.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:47.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:27:47.848 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:27:47.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:47.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:27:48.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:48.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:48.374+0000 7faea4bda7c0 -1 osd.1 38 log_to_monitors true 2024-06-11T18:27:49.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:49.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:49.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:27:49.251 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:27:49.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:49.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:27:49.715 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 39 last_clean_interval [10,38) [v2:127.0.0.1:6810/141153708,v1:127.0.0.1:6811/141153708] [v2:127.0.0.1:6812/141153708,v1:127.0.0.1:6813/141153708] exists,up 0d538a17-0d08-4f0d-811d-920ed4ce207c 2024-06-11T18:27:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:27:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:27:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:27:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:27:49.716 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:27:49.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:27:49.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:27:49.719 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:27:49.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:27:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:27:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:27:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:27:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:27:49.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:27:49.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:27:49.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:27:49.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:27:49.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:27:49.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:27:49.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:27:49.740 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:49.737+0000 7fba185677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:49.742 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:49.737+0000 7fba185677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:49.743 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:49.741+0000 7fba185677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:50.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:50.292 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:50.289+0000 7fba185677c0 -1 Falling back to public interface 2024-06-11T18:27:50.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:51.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:51.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:51.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:27:51.570 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:27:51.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:51.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:51.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:52.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:52.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:52.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:27:52.896 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:27:52.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:52.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:53.155 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:27:53.153+0000 7fba185677c0 -1 osd.2 40 log_to_monitors true 2024-06-11T18:27:53.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:27:54.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:27:54.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:27:54.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:27:54.234 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:27:54.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:27:54.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 49 up_thru 49 down_at 41 last_clean_interval [15,40) [v2:127.0.0.1:6818/1676745713,v1:127.0.0.1:6819/1676745713] [v2:127.0.0.1:6820/1676745713,v1:127.0.0.1:6821/1676745713] exists,up 816a2b1b-37a4-4ef9-9f9b-2722aa607e94 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-11T18:27:54.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:27:55.571 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' removed 2024-06-11T18:27:55.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:27:55.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:27:56.608 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:27:56.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:27:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:28:00.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-11T18:28:00.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:28:00.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:28:00.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:28:00.153 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:28:00.154 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:28:00.154 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:28:00.154 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:28:00.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:28:00.155 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:28:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:28:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:28:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:28:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:28:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:28:00.294 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:28:00.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:28:00.646 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:28:00.646 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:28:00.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:28:00.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:28:00.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:28:00.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=184683593733 2024-06-11T18:28:00.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 184683593733 2024-06-11T18:28:00.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593733' 2024-06-11T18:28:00.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:28:00.840 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:28:00.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495620 2024-06-11T18:28:00.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495620 2024-06-11T18:28:00.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593733 1-197568495620' 2024-06-11T18:28:00.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:28:00.994 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:28:01.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397507 2024-06-11T18:28:01.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397507 2024-06-11T18:28:01.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593733 1-197568495620 2-210453397507' 2024-06-11T18:28:01.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:28:01.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-184683593733 2024-06-11T18:28:01.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:28:01.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:28:01.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-184683593733 2024-06-11T18:28:01.140 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:28:01.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=184683593733 2024-06-11T18:28:01.141 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 184683593733' 2024-06-11T18:28:01.141 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 184683593733 2024-06-11T18:28:01.142 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:28:01.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593732 -lt 184683593733 2024-06-11T18:28:01.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:28:02.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:28:02.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:28:02.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593733 -lt 184683593733 2024-06-11T18:28:02.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:28:02.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-197568495620 2024-06-11T18:28:02.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:28:02.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:28:02.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-197568495620 2024-06-11T18:28:02.801 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:28:02.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495620 2024-06-11T18:28:02.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 197568495620' 2024-06-11T18:28:02.803 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 197568495620 2024-06-11T18:28:02.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:28:03.162 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495620 -lt 197568495620 2024-06-11T18:28:03.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:28:03.163 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-210453397507 2024-06-11T18:28:03.164 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:28:03.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:28:03.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-210453397507 2024-06-11T18:28:03.168 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:28:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397507 2024-06-11T18:28:03.169 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-06-11T18:28:03.169 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 210453397507 2024-06-11T18:28:03.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:28:03.540 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397507 -lt 210453397507 2024-06-11T18:28:03.541 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:28:03.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:28:03.542 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:28:03.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:28:03.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:28:03.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:28:03.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:28:03.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:28:03.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:28:03.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:28:03.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:28:04.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:28:04.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:28:04.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:28:04.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:28:04.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:28:04.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:28:04.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:28:04.718 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-11T18:28:04.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:28:04.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-11T18:28:04.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-11T18:28:04.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:28:05.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-11T18:28:05.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-11T18:28:05.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:28:05.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:28:05.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:28:05.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:28:05.098 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:28:05.099 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:28:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:28:05.719 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:28:05.736 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:28:05.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:28:05.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:28:05.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:28:05.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:28:05.737 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:28:06.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:27:19.289216+0000 '>' 2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:06.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:28:07.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:28:07.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:28:07.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:28:07.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:28:07.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:28:07.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:28:07.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:28:07.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:27:19.289216+0000 '>' 2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:07.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:28:08.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:28:08.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:28:08.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:28:08.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:28:08.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:28:08.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:28:08.392 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:28:08.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:27:19.289216+0000 '>' 2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:08.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:28:09.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:28:09.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:28:09.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:28:09.728 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:28:09.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:28:09.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:28:09.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:28:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:28:06.148602+0000 '>' 2024-06-11T18:27:19.289216+0000 2024-06-11T18:28:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:28:10.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-11T18:28:10.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-11T18:28:10.064 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-11T18:28:10.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-11T18:28:10.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-11T18:28:10.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:28:10.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:28:10.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:28:10.236 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:28:10.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:28:10.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:28:10.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:28:10.547 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:28:10.548 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:28:10.549 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:28:10.550 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:28:10.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:28:10.551 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:28:10.552 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:28:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:28:10.553 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:28:10.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:28:10.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:28:10.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:28:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:28:10.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:28:10.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:28:10.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:10.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:10.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:28:10.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:28:10.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:28:10.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:28:10.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:28:10.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:28:10.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:28:10.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:28:10.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:28:10.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:28:10.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:28:10.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:28:10.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:28:10.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:28:10.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:28:10.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:28:10.596 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:28:10.596 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:28:10.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:28:10.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:28:10.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:28:10.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:28:10.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:10.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:10.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:28:10.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:28:10.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:28:10.603 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:28:10.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:28:10.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:10.605 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:10.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:28:10.607 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-11T18:28:10.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:28:10.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:28:11.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:28:11.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:28:11.147 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:28:11.148 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:28:11.148 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:11.148 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:11.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:11.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:28:11.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:28:11.401 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:28:11.401 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:28:11.401 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:28:11.402 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:28:11.402 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:28:11.403 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:28:11.403 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:28:11.403 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:28:11.403 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:28:11.403 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:11.404 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:11.404 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:28:11.404 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:28:11.404 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:28:11.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:28:11.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:28:11.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:28:11.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:28:11.525 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:28:11.526 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:28:11.526 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:28:11.526 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:28:11.526 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:28:11.526 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:11.527 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:11.527 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:28:11.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:28:11.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:28:11.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-11T18:28:11.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:28:11.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:28:11.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:28:11.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:28:11.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:28:11.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:28:12.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:28:12.585 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:28:12.585 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:28:12.586 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:28:12.586 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:12.586 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:12.586 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:12.587 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:28:12.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:28:12.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-11T18:28:12.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:28:12.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-11T18:28:12.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:28:12.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:28:12.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:28:12.629 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:12.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:12.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:28:12.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:28:12.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=465e004d-d7ec-436b-a32b-a218dcace33a 2024-06-11T18:28:12.633 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 465e004d-d7ec-436b-a32b-a218dcace33a 2024-06-11T18:28:12.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 465e004d-d7ec-436b-a32b-a218dcace33a' 2024-06-11T18:28:12.633 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:28:12.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8l2hmIIq2JhAAgmSBkYjwfiWtUOCAOtKumQ== 2024-06-11T18:28:12.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8l2hmIIq2JhAAgmSBkYjwfiWtUOCAOtKumQ=="}' 2024-06-11T18:28:12.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 465e004d-d7ec-436b-a32b-a218dcace33a -i td/osd-scrub-repair/0/new.json 2024-06-11T18:28:12.919 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:28:12.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:28:12.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8l2hmIIq2JhAAgmSBkYjwfiWtUOCAOtKumQ== --osd-uuid 465e004d-d7ec-436b-a32b-a218dcace33a 2024-06-11T18:28:12.972 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:12.969+0000 7f2872d4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:13.004 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.001+0000 7f2872d4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:13.005 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.001+0000 7f2872d4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:13.006 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.001+0000 7f2872d4a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:28:13.006 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.001+0000 7f2872d4a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:28:13.006 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.001+0000 7f2872d4a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:28:13.111 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:13.109+0000 7f2872d4a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:28:20.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:28:20.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:28:20.743 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:28:20.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:28:20.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:28:21.151 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:28:21.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:28:21.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:28:21.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:28:21.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:28:21.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:28:21.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:21.181+0000 7f324e7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:21.227 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:21.225+0000 7f324e7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:21.229 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:21.225+0000 7f324e7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:21.470 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:28:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:28:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:21.796 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:21.793+0000 7f324e7b47c0 -1 Falling back to public interface 2024-06-11T18:28:21.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:22.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:22.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:22.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:28:22.799 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:28:22.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:22.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:23.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:24.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:24.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:24.136 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:28:24.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:28:24.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:24.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:24.756 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:24.753+0000 7f324e7b47c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:28:25.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:25.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:25.478 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:28:25.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:28:25.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:25.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:25.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:26.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:26.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:26.912 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:28:26.912 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:28:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:26.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:27.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:28.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:28.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:28.327 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:28:28.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:28:28.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:28.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:28.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:29.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:29.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:29.657 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:28:29.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:28:29.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:29.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:30.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:31.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:31.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:31.105 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:28:31.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:28:31.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:31.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:31.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:32.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:32.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:32.443 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:28:32.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:28:32.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:32.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:32.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:33.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:33.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:33.901 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:28:33.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:28:33.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:33.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:34.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:35.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:35.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:35.236 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:28:35.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:28:35.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:35.237 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:35.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:36.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:36.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:36.614 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:28:36.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:28:36.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:36.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:37.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:38.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:38.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:38.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:28:38.078 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:28:38.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:38.079 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:38.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:39.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:39.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:39.570 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:28:39.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:28:39.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:39.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:40.191 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:41.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:41.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:41.193 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:28:41.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:28:41.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:41.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:41.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:42.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:42.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:42.625 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:28:42.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:28:42.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:42.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:43.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:44.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:44.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:44.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:28:44.116 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:28:44.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:44.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:44.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:45.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:45.784 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:28:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:28:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:46.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:47.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:47.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:47.770 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:28:47.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:28:47.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:47.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:48.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:49.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:49.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:49.515 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:28:49.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:28:49.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:49.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:50.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:51.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:51.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:51.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:28:51.329 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:28:51.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:51.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:52.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:53.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:53.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:53.130 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:28:53.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:28:53.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:53.131 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:53.836 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:54.838 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:54.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:54.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:28:54.839 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:28:54.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:54.839 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:55.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:28:56.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:28:56.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:28:56.575 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:28:56.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:28:56.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:28:56.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:28:56.968 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4081715120,v1:127.0.0.1:6803/4081715120] [v2:127.0.0.1:6804/4081715120,v1:127.0.0.1:6805/4081715120] exists,up 465e004d-d7ec-436b-a32b-a218dcace33a 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:28:56.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:28:56.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:28:56.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:28:56.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:28:56.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:28:56.976 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:28:56.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28d06788-1dc9-41be-8b35-42ac3af94ec9 2024-06-11T18:28:56.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 28d06788-1dc9-41be-8b35-42ac3af94ec9' 2024-06-11T18:28:56.977 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 28d06788-1dc9-41be-8b35-42ac3af94ec9 2024-06-11T18:28:56.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:28:56.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBol2hmNs+AOxAA4go9M2/+reXdZ4GSp2nx1w== 2024-06-11T18:28:57.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBol2hmNs+AOxAA4go9M2/+reXdZ4GSp2nx1w=="}' 2024-06-11T18:28:57.000 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28d06788-1dc9-41be-8b35-42ac3af94ec9 -i td/osd-scrub-repair/1/new.json 2024-06-11T18:28:57.443 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:28:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:28:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBol2hmNs+AOxAA4go9M2/+reXdZ4GSp2nx1w== --osd-uuid 28d06788-1dc9-41be-8b35-42ac3af94ec9 2024-06-11T18:28:57.522 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.517+0000 7f77cd2137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.709+0000 7f77cd2137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.709+0000 7f77cd2137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:28:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.713+0000 7f77cd2137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:28:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.713+0000 7f77cd2137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:28:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.713+0000 7f77cd2137c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:28:57.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:28:57.809+0000 7f77cd2137c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:29:05.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:29:05.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:29:05.791 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:29:05.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:29:05.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:29:06.205 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:29:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:29:06.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:29:06.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:29:06.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:29:06.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:29:06.240 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:06.237+0000 7f430d9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:06.274 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:06.269+0000 7f430d9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:06.275 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:06.273+0000 7f430d9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:06.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:29:06.522 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:29:06.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:06.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:06.833 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:06.829+0000 7f430d9337c0 -1 Falling back to public interface 2024-06-11T18:29:06.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:07.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:07.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:07.871 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:29:07.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:29:07.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:07.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:08.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:09.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:09.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:09.209 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:29:09.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:29:09.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:09.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:09.543 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:09.687 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:09.685+0000 7f430d9337c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:29:10.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:10.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:10.545 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:29:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:29:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:10.546 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:11.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:12.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:12.007 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:12.007 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:29:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:29:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:12.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:13.330 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:13.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:13.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:29:13.331 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:29:13.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:13.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:13.712 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:14.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:14.714 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:14.715 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:29:14.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:29:14.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:14.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:15.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:16.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:16.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:16.176 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:29:16.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:29:16.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:16.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:17.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:17.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:17.593 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:29:17.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:29:17.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:17.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:18.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:19.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:19.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:19.092 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:29:19.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:29:19.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:19.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:19.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:20.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:20.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:20.429 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:29:20.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:29:20.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:20.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:20.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:21.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:29:21.805 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:29:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:22.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:23.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:23.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:23.265 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:29:23.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:29:23.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:23.266 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:23.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:24.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:24.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:24.601 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:29:24.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:29:24.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:24.602 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:25.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:26.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:26.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:26.070 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:29:26.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:29:26.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:26.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:27.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:27.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:27.428 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:29:27.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:29:27.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:27.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:27.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:28.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:28.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:28.895 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:29:28.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:29:28.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:28.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:29.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:30.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:30.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:30.321 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:29:30.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:29:30.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:30.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:31.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:32.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:32.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:32.068 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:29:32.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:29:32.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:32.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:32.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:33.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:33.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:33.527 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:29:33.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:29:33.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:33.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:34.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:35.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:35.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:29:35.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:29:35.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:35.161 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:36.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:37.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:37.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:37.359 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:29:37.360 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:29:37.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:37.360 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:37.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:38.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:38.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:38.966 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:29:38.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:29:38.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:38.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:39.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:40.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:40.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:40.536 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:29:40.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:29:40.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:40.537 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:41.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:42.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:42.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:42.103 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:29:42.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:29:42.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:42.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:29:42.527 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3945369803,v1:127.0.0.1:6811/3945369803] [v2:127.0.0.1:6812/3945369803,v1:127.0.0.1:6813/3945369803] exists,up 28d06788-1dc9-41be-8b35-42ac3af94ec9 2024-06-11T18:29:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:29:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:29:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:29:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:29:42.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:29:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 2024-06-11T18:29:42.534 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 2024-06-11T18:29:42.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1' 2024-06-11T18:29:42.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:29:42.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWl2hmVbg+IRAA5zj/Am8cnk9hc3rsQRpqlQ== 2024-06-11T18:29:42.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWl2hmVbg+IRAA5zj/Am8cnk9hc3rsQRpqlQ=="}' 2024-06-11T18:29:42.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 -i td/osd-scrub-repair/2/new.json 2024-06-11T18:29:43.095 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:29:43.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T18:29:43.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWl2hmVbg+IRAA5zj/Am8cnk9hc3rsQRpqlQ== --osd-uuid 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 2024-06-11T18:29:43.156 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.153+0000 7faa4f2187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:43.230 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.225+0000 7faa4f2187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.229+0000 7faa4f2187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.229+0000 7faa4f2187c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:29:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.229+0000 7faa4f2187c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:29:43.232 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.229+0000 7faa4f2187c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:29:43.323 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:43.321+0000 7faa4f2187c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T18:29:51.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T18:29:51.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:29:51.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T18:29:51.498 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T18:29:51.498 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:29:51.940 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T18:29:51.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T18:29:51.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:29:51.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:29:51.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:29:51.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:29:51.974 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:51.969+0000 7f6e399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:52.009 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:52.005+0000 7f6e399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:52.011 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:52.009+0000 7f6e399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:29:52.326 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:29:52.327 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:29:52.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:52.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:52.573 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:52.569+0000 7f6e399b97c0 -1 Falling back to public interface 2024-06-11T18:29:52.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:53.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:53.706 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:53.706 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:29:53.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:29:53.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:53.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:54.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:55.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:55.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:55.045 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:29:55.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:29:55.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:55.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:55.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:55.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:29:55.469+0000 7f6e399b97c0 -1 osd.2 0 log_to_monitors true 2024-06-11T18:29:56.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:56.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:56.374 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:29:56.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:29:56.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:56.375 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:56.794 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:57.798 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:29:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:29:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:58.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:29:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:29:59.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:29:59.152 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:29:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:29:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:29:59.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:29:59.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:00.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:00.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:00.520 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:30:00.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:30:00.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:00.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:00.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:01.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:01.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:01.931 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:30:01.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:30:01.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:01.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:02.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:03.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:03.417 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:03.417 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:30:03.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:30:03.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:03.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:03.928 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:04.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:04.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:04.931 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:30:04.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:30:04.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:04.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:05.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:06.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:06.262 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:06.262 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:30:06.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:30:06.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:06.263 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:07.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:07.802 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:07.802 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:30:07.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:30:07.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:07.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:08.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:09.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:09.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:09.356 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:30:09.356 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:30:09.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:09.357 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:09.907 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:10.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:10.909 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:10.909 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:30:10.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:30:10.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:10.910 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:11.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:12.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:12.243 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:30:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:30:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:12.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:12.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:13.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:13.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:13.572 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:30:13.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:30:13.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:13.573 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:14.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:15.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:15.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:15.129 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:30:15.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:30:15.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:15.130 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:15.687 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:16.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:16.689 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:16.689 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:30:16.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:30:16.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:16.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:17.345 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:18.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:18.347 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:18.347 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:30:18.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:30:18.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:18.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:18.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:19.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:19.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:19.749 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:30:19.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:30:19.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:19.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:20.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:21.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:21.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:21.155 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:30:21.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:30:21.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:21.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:21.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:22.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:22.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:22.746 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:30:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:30:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:23.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:24.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:24.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:24.423 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:30:24.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:30:24.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:24.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:26.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:26.474 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:30:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:30:26.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:26.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:27.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:30:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:30:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:30:28.020 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:30:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:30:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:30:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2404064825,v1:127.0.0.1:6819/2404064825] [v2:127.0.0.1:6820/2404064825,v1:127.0.0.1:6821/2404064825] exists,up 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-11T18:30:28.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:30:28.781 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:30:28.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:30:28.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:30:29.254 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:30:29.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:30:30.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:30:34.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-11T18:30:34.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:30:34.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:30:34.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:30:34.107 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:30:34.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:30:34.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:30:34.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:30:34.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:30:34.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:30:34.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:30:34.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:30:34.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:30:34.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:30:34.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:30:34.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:30:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:30:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:30:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:30:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:30:34.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:34.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:30:34.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-06-11T18:30:34.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-06-11T18:30:34.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-06-11T18:30:34.773 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:34.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:30:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-06-11T18:30:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-06-11T18:30:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-06-11T18:30:34.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:34.921 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:30:35.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-06-11T18:30:35.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-06-11T18:30:35.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509448' 2024-06-11T18:30:35.065 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:35.065 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-06-11T18:30:35.066 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:35.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:30:35.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-06-11T18:30:35.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:35.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-06-11T18:30:35.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-11T18:30:35.069 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836506 2024-06-11T18:30:35.070 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:30:35.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836506 2024-06-11T18:30:35.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:30:36.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:30:36.389 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:30:36.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-11T18:30:36.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:36.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-06-11T18:30:36.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:36.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:30:36.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-06-11T18:30:36.724 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-06-11T18:30:36.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-11T18:30:36.727 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672977 2024-06-11T18:30:36.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:30:37.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-06-11T18:30:37.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:37.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-06-11T18:30:37.102 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:37.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:30:37.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-06-11T18:30:37.105 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:37.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-06-11T18:30:37.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-11T18:30:37.107 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509448 2024-06-11T18:30:37.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:30:37.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-11T18:30:37.478 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:30:37.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:30:37.479 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:30:37.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:30:37.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:30:37.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:30:37.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:30:37.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:30:37.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:30:37.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:30:37.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:30:38.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:30:38.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:30:38.297 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:30:38.298 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:30:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:30:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:30:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:30:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-11T18:30:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=ecpool 2024-06-11T18:30:38.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T18:30:38.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-06-11T18:30:38.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T18:30:38.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-11T18:30:39.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-11T18:30:39.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-11T18:30:40.212 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'ecpool' created 2024-06-11T18:30:40.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:30:41.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-06-11T18:30:41.236 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-11T18:30:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 2 allow_ec_overwrites to true 2024-06-11T18:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T18:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:30:41.803 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:30:41.803 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:30:41.803 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:30:41.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:30:41.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:30:41.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:30:41.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:30:41.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:30:41.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:30:41.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:30:41.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:30:41.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:42.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-06-11T18:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-06-11T18:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-06-11T18:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-06-11T18:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-06-11T18:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672979' 2024-06-11T18:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:30:42.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-06-11T18:30:42.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-06-11T18:30:42.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672979 2-64424509450' 2024-06-11T18:30:42.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:42.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-06-11T18:30:42.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:42.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:30:42.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-06-11T18:30:42.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:42.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-06-11T18:30:42.734 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836508 2024-06-11T18:30:42.734 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-11T18:30:42.735 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:30:43.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836508 2024-06-11T18:30:43.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:30:44.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:30:44.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:30:44.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836508 2024-06-11T18:30:44.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:44.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-06-11T18:30:44.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:44.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:30:44.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-06-11T18:30:44.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:44.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-06-11T18:30:44.395 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672979 2024-06-11T18:30:44.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-11T18:30:44.396 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:30:44.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672979 2024-06-11T18:30:44.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:30:44.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-06-11T18:30:44.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:30:44.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:30:44.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:30:44.772 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-06-11T18:30:44.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-06-11T18:30:44.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-11T18:30:44.774 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509450 2024-06-11T18:30:44.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:30:45.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-11T18:30:45.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:30:45.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:30:45.176 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:30:45.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:30:45.608 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:30:45.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:30:45.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:30:45.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:30:45.609 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:30:45.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:30:45.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:30:45.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:30:45.948 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:30:45.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:30:45.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-11T18:30:46.350 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-11T18:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:30:46.912 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:30:46.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:30:47.422 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:30:47.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:30:47.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:30:47.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:30:47.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-11T18:30:47.849 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T18:30:47.850 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-11T18:30:47.850 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-11T18:30:47.851 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-11T18:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-11T18:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-11T18:30:48.701 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is set 2024-06-11T18:30:48.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-11T18:30:49.195 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:30:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:30:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:30:49.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:30:49.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-11T18:30:49.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-11T18:30:49.603 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-11T18:30:49.604 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-11T18:30:49.604 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-11T18:30:49.943 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-11T18:30:49.943 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:30:49.944 INFO:tasks.workunit.client.0.smithi012.stderr:0' 2024-06-11T18:30:49.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-11T18:30:49.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-11T18:30:49.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-11T18:30:49.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-11T18:30:49.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:30:49.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:30:49.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:30:49.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:30:49.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:30:49.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:30:49.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:30:51.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:30:51.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:30:51.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:30:51.260 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:30:51.261 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:30:51.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:30:51.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:30:51.261 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:30:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:30:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:30:52.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-11T18:30:52.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:30:52.579 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:30:52.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:30:52.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:30:52.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:30:53.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:30:53.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-11T18:30:53.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-11T18:30:55.559 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-11T18:30:56.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-11T18:30:56.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-11T18:30:57.745 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-11T18:30:58.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-11T18:30:58.282 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-11T18:30:59.343 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-11T18:30:59.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-11T18:30:59.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-11T18:31:01.480 INFO:tasks.workunit.client.0.smithi012.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-11T18:31:02.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:31:02.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:02.017 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:31:02.019 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:31:02.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:31:02.021 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:31:02.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:31:02.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:31:02.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:31:02.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:31:02.055 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:02.052+0000 7fdd3e6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:02.058 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:02.056+0000 7fdd3e6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:02.061 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:02.056+0000 7fdd3e6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:02.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:31:02.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:31:02.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:31:02.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:31:02.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:31:02.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:02.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:31:02.353 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:31:02.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:02.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:02.597 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:02.592+0000 7fdd3e6e97c0 -1 Falling back to public interface 2024-06-11T18:31:02.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:03.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:03.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:03.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:31:03.724 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:31:03.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:03.725 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:04.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:05.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:05.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:05.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:31:05.058 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:31:05.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:05.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:05.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:05.517 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:05.512+0000 7fdd3e6e97c0 -1 osd.0 34 log_to_monitors true 2024-06-11T18:31:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:31:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:31:06.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:06.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:06.742 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 41 up_thru 18 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/1092039196,v1:127.0.0.1:6803/1092039196] [v2:127.0.0.1:6804/1092039196,v1:127.0.0.1:6805/1092039196] exists,up 465e004d-d7ec-436b-a32b-a218dcace33a 2024-06-11T18:31:06.742 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:31:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:06.744 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:31:06.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:31:06.747 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.1 2024-06-11T18:31:06.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:31:06.748 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:31:06.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:31:06.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:31:06.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:31:06.749 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:31:06.784 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:06.780+0000 7fe5db4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:06.856 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:06.852+0000 7fe5db4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:06.858 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:06.856+0000 7fe5db4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:31:07.099 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:31:07.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:07.100 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:31:07.397 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:07.392+0000 7fe5db4857c0 -1 Falling back to public interface 2024-06-11T18:31:07.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:08.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:08.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:08.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:31:08.427 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:31:08.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:08.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:31:08.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:09.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:09.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:09.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:31:09.757 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:31:09.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:09.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:31:10.082 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:10.373 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:10.368+0000 7fe5db4857c0 -1 osd.1 36 log_to_monitors true 2024-06-11T18:31:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:31:11.085 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:31:11.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:11.086 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1266596554,v1:127.0.0.1:6811/1266596554] [v2:127.0.0.1:6812/1266596554,v1:127.0.0.1:6813/1266596554] exists,up 28d06788-1dc9-41be-8b35-42ac3af94ec9 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:31:11.513 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:31:11.514 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:11.515 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:11.516 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:11.516 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:11.516 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-11T18:31:11.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:31:11.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-11T18:31:11.518 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.2 2024-06-11T18:31:11.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-11T18:31:11.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-11T18:31:11.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-11T18:31:11.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:31:11.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:31:11.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:31:11.543 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:11.540+0000 7fad125287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:11.545 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:11.540+0000 7fad125287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:11.547 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:11.544+0000 7fad125287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:12.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-11T18:31:12.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:31:12.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:31:12.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:12.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:31:12.100 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:12.096+0000 7fad125287c0 -1 Falling back to public interface 2024-06-11T18:31:12.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:13.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:13.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:13.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:31:13.389 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:31:13.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:13.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:31:13.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:14.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:31:15.019 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:15.016+0000 7fad125287c0 -1 osd.2 38 log_to_monitors true 2024-06-11T18:31:15.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:16.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:16.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:16.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:31:16.069 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:31:16.069 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:16.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:osd.2 up in weight 1 up_from 47 up_thru 37 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/3180901619,v1:127.0.0.1:6819/3180901619] [v2:127.0.0.1:6820/3180901619,v1:127.0.0.1:6821/3180901619] exists,up 6a3dda3d-e692-4da7-8c04-f9c83d82c2e1 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-11T18:31:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:31:17.638 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' removed 2024-06-11T18:31:17.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:31:17.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:31:18.714 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:31:18.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:31:19.739 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:31:22.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-11T18:31:22.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:31:22.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:31:22.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:31:22.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:31:22.121 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:31:22.121 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:31:22.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:31:22.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:31:22.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:31:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:31:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:31:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:31:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:31:22.265 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:31:22.266 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:31:22.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:31:22.612 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:31:22.612 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:31:22.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:31:22.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:31:22.612 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:31:22.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=176093659141 2024-06-11T18:31:22.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 176093659141 2024-06-11T18:31:22.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-176093659141' 2024-06-11T18:31:22.769 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:31:22.770 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:31:22.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561028 2024-06-11T18:31:22.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561028 2024-06-11T18:31:22.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-176093659141 1-188978561028' 2024-06-11T18:31:22.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:31:22.925 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:31:23.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462915 2024-06-11T18:31:23.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462915 2024-06-11T18:31:23.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-176093659141 1-188978561028 2-201863462915' 2024-06-11T18:31:23.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:31:23.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-176093659141 2024-06-11T18:31:23.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:31:23.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:31:23.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-176093659141 2024-06-11T18:31:23.076 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:31:23.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=176093659141 2024-06-11T18:31:23.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 176093659141' 2024-06-11T18:31:23.078 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 176093659141 2024-06-11T18:31:23.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:31:23.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659140 -lt 176093659141 2024-06-11T18:31:23.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:31:24.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:31:24.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:31:24.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659141 -lt 176093659141 2024-06-11T18:31:24.735 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:31:24.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-188978561028 2024-06-11T18:31:24.736 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:31:24.737 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:31:24.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-188978561028 2024-06-11T18:31:24.738 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:31:24.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561028 2024-06-11T18:31:24.740 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-06-11T18:31:24.740 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 188978561028 2024-06-11T18:31:24.740 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:31:25.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561028 -lt 188978561028 2024-06-11T18:31:25.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:31:25.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:31:25.114 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-201863462915 2024-06-11T18:31:25.115 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:31:25.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-201863462915 2024-06-11T18:31:25.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:31:25.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462915 2024-06-11T18:31:25.119 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 201863462915 2024-06-11T18:31:25.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-06-11T18:31:25.119 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:31:25.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462915 -lt 201863462915 2024-06-11T18:31:25.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:31:25.493 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:31:25.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:31:25.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:31:25.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:31:25.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:31:26.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:31:26.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:31:26.320 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:31:26.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:31:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:31:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:31:26.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:31:26.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-11T18:31:26.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-11T18:31:26.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-11T18:31:26.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-11T18:31:26.696 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:31:27.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-11T18:31:27.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-11T18:31:27.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-06-11T18:31:27.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-06-11T18:31:27.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:31:27.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:31:27.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:31:27.043 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:31:27.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:27.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-06-11T18:31:27.703 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-11T18:31:27.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:31:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:31:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:31:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:31:27.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:31:27.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:31:27.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:31:27.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:31:27.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:31:28.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:30:40.105630+0000 '>' 2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:28.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:31:29.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:31:29.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:31:29.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:31:29.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:31:29.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:31:29.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:31:29.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:31:29.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:30:40.105630+0000 '>' 2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:29.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:31:30.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:31:30.384 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:31:30.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:31:30.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:31:30.385 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:31:30.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:31:30.386 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:31:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:30:40.105630+0000 '>' 2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:30.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:31:31.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:31:31.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:31:31.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:31:31.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:31:31.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:31:31.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:31:31.723 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:31:32.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:31:28.065530+0000 '>' 2024-06-11T18:30:40.105630+0000 2024-06-11T18:31:32.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:31:32.062 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-11T18:31:32.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-11T18:31:32.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-11T18:31:32.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-11T18:31:32.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-11T18:31:32.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-11T18:31:32.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-11T18:31:32.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-11T18:31:32.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-11T18:31:32.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:31:32.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:31:32.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:31:32.518 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:31:32.519 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:31:32.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:31:32.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:31:32.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:31:32.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:31:32.522 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:31:32.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:31:32.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:31:32.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:31:32.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:31:32.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:31:32.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:31:32.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:31:32.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:31:32.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:32.554 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:32.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:31:32.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:31:32.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:31:32.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:31:32.556 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:31:32.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:31:32.557 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:31:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:31:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:31:32.557 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:31:32.560 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:31:32.560 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:31:32.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:31:32.562 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:31:32.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:31:32.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:31:32.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:31:32.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:31:32.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:31:32.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:31:32.566 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:31:32.567 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:31:32.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:31:32.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:31:32.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:31:32.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:32.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:32.570 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:31:32.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:31:32.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:31:32.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:31:32.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:31:32.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:32.574 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:32.574 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:31:32.576 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:31:32.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:31:32.577 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:31:32.578 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-11T18:31:33.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:31:33.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:33.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:33.079 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:33.079 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:33.079 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:33.079 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:33.080 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-11T18:31:33.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:31:33.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:31:33.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:31:33.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:31:33.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:31:33.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:31:33.327 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:31:33.327 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:31:33.327 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:31:33.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:31:33.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:31:33.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:31:33.447 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:31:33.448 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:33.449 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:33.450 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:31:33.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:31:33.450 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:31:33.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:31:34.489 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:31:34.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:34.886 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:34.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:34.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:34.887 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:34.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:34.891 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:31:34.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:31:34.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:31:34.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:31:34.893 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:31:34.894 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff2b5afa-537a-44fb-9ab9-8e67129672f2 2024-06-11T18:31:34.895 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 ff2b5afa-537a-44fb-9ab9-8e67129672f2 2024-06-11T18:31:34.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff2b5afa-537a-44fb-9ab9-8e67129672f2' 2024-06-11T18:31:34.896 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:31:34.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGmGhmU+/ZIBAA3p76WbYNPzsdKYBiHwkAqA== 2024-06-11T18:31:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGmGhmU+/ZIBAA3p76WbYNPzsdKYBiHwkAqA=="}' 2024-06-11T18:31:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff2b5afa-537a-44fb-9ab9-8e67129672f2 -i td/osd-scrub-repair/0/new.json 2024-06-11T18:31:34.897 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:31:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:31:34.897 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAGmGhmU+/ZIBAA3p76WbYNPzsdKYBiHwkAqA== --osd-uuid ff2b5afa-537a-44fb-9ab9-8e67129672f2 2024-06-11T18:31:34.907 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.904+0000 7f47e7c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:34.935 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.932+0000 7f47e7c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:34.937 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.932+0000 7f47e7c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:34.937 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.932+0000 7f47e7c4b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:31:34.937 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.932+0000 7f47e7c4b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:31:34.938 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:34.932+0000 7f47e7c4b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:31:35.043 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:35.040+0000 7f47e7c4b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:31:42.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:31:42.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:31:42.837 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:31:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:31:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:31:43.250 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:31:43.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:31:43.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:31:43.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:31:43.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:31:43.251 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:31:43.286 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:43.280+0000 7f22497497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:43.329 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:43.324+0000 7f22497497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:43.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:43.328+0000 7f22497497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:31:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:43.564 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:31:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:31:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:43.868 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:43.912 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:43.908+0000 7f22497497c0 -1 Falling back to public interface 2024-06-11T18:31:44.869 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:44.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:44.870 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:31:44.870 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:31:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:45.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:46.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:46.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:46.196 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:31:46.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:31:46.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:46.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:46.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:46.833 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:31:46.828+0000 7f22497497c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:31:47.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:47.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:47.531 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:31:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:31:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:47.915 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:48.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:48.916 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:48.917 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:31:48.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:31:48.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:48.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:49.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:50.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:50.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:50.255 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:31:50.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:31:50.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:50.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:50.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:51.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:51.680 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:51.681 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:31:51.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:31:51.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:51.681 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:52.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:53.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:53.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:53.120 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:31:53.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:31:53.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:53.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:53.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:54.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:54.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:54.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:31:54.465 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:31:54.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:54.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:54.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:55.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:55.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:55.906 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:31:55.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:31:55.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:55.906 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:56.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:57.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:57.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:57.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:31:57.244 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:31:57.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:57.245 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:31:58.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:31:58.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:31:58.614 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:31:58.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:31:58.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:31:58.615 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:31:59.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:00.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:00.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:00.107 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:32:00.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:32:00.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:00.108 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:00.438 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:01.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:01.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:01.440 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:32:01.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:32:01.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:01.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:01.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:02.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:02.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:02.787 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:32:02.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:32:02.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:02.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:03.117 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:04.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:04.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:04.120 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:32:04.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:32:04.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:04.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:04.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:05.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:05.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:05.450 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:32:05.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:32:05.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:05.451 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:05.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:06.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:06.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:06.864 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:32:06.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:32:06.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:06.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:07.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:08.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:08.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:08.589 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:32:08.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:32:08.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:08.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:09.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:10.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:10.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:10.092 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:32:10.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:32:10.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:10.092 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:11.152 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:12.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:12.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:12.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:32:12.154 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:32:12.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:12.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:13.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:13.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:13.989 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:32:13.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:32:13.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:13.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:15.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:16.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:16.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:16.048 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:32:16.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:32:16.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:16.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:16.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:17.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:17.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:17.789 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:32:17.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:32:17.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:17.790 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:18.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:19.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:19.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:19.159 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:32:19.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:32:19.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:19.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:32:19.529 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3434316030,v1:127.0.0.1:6803/3434316030] [v2:127.0.0.1:6804/3434316030,v1:127.0.0.1:6805/3434316030] exists,up ff2b5afa-537a-44fb-9ab9-8e67129672f2 2024-06-11T18:32:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:32:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:32:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:32:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:32:19.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:32:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:32:19.533 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-11T18:32:19.534 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:32:19.535 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:32:19.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05116241-6da0-471d-9680-d6010422a9ce 2024-06-11T18:32:19.536 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 05116241-6da0-471d-9680-d6010422a9ce' 2024-06-11T18:32:19.536 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 05116241-6da0-471d-9680-d6010422a9ce 2024-06-11T18:32:19.537 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:32:19.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzmGhmyONZIRAAzOVCejLOerVGzZc3hv7bQA== 2024-06-11T18:32:19.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzmGhmyONZIRAAzOVCejLOerVGzZc3hv7bQA=="}' 2024-06-11T18:32:19.561 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05116241-6da0-471d-9680-d6010422a9ce -i td/osd-scrub-repair/1/new.json 2024-06-11T18:32:19.985 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:32:20.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:32:20.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAzmGhmyONZIRAAzOVCejLOerVGzZc3hv7bQA== --osd-uuid 05116241-6da0-471d-9680-d6010422a9ce 2024-06-11T18:32:20.044 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.039+0000 7fb321e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:20.146 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.139+0000 7fb321e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:20.148 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.143+0000 7fb321e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:20.149 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.143+0000 7fb321e647c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:32:20.149 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.143+0000 7fb321e647c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:32:20.149 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.143+0000 7fb321e647c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:32:20.224 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:20.219+0000 7fb321e647c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:32:28.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:32:28.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:32:28.727 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:32:28.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:32:28.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:32:29.134 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:32:29.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:32:29.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:32:29.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:32:29.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:32:29.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:32:29.170 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:29.163+0000 7ff262fda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:29.208 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:29.203+0000 7ff262fda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:29.210 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:29.207+0000 7ff262fda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:32:29.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:29.445 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:32:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:32:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:29.446 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:29.773 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:29.767+0000 7ff262fda7c0 -1 Falling back to public interface 2024-06-11T18:32:29.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:30.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:30.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:30.829 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:32:30.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:32:30.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:30.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:31.155 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:32.157 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:32.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:32.158 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:32:32.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:32:32.158 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:32.159 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:32.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:32.646 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:32:32.643+0000 7ff262fda7c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:32:33.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:33.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:33.494 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:32:33.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:32:33.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:33.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:33.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:34.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:34.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:34.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:32:34.955 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:32:34.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:34.955 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:35.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:36.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:36.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:36.405 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:32:36.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:32:36.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:36.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:36.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:37.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:37.898 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:37.899 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:32:37.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:32:37.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:37.899 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:38.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:39.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:39.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:39.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:32:39.276 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:32:39.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:39.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:40.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:40.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:40.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:32:40.611 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:32:40.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:40.612 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:41.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:42.076 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:32:42.077 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:32:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:42.400 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:43.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:43.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:43.403 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:32:43.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:32:43.403 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:43.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:44.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:44.798 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:44.798 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:32:44.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:32:44.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:44.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:45.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:46.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:46.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:46.232 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:32:46.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:32:46.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:46.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:46.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:47.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:47.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:47.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:32:47.476 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:32:47.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:47.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:47.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:48.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:48.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:48.812 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:32:48.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:32:48.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:48.813 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:49.151 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:50.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:50.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:50.153 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:32:50.154 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:32:50.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:50.154 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:50.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:51.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:51.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:51.491 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:32:51.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:32:51.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:51.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:52.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:53.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:53.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:53.149 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:32:53.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:32:53.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:53.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:53.741 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:54.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:54.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:54.743 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:32:54.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:32:54.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:54.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:55.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:56.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:56.209 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:56.209 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:32:56.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:32:56.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:56.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:57.090 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:32:58.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:32:58.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:32:58.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:32:58.094 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:32:58.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:32:58.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:32:59.290 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:00.291 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:00.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:00.292 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:33:00.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:33:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:00.293 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:33:00.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:01.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:01.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:01.990 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:33:01.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:33:01.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:01.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:33:02.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:03.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:03.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:03.697 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:33:03.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:33:03.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:03.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:33:04.514 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:05.515 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:05.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:05.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:33:05.516 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:33:05.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:05.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:33:05.881 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2547567379,v1:127.0.0.1:6811/2547567379] [v2:127.0.0.1:6812/2547567379,v1:127.0.0.1:6813/2547567379] exists,up 05116241-6da0-471d-9680-d6010422a9ce 2024-06-11T18:33:05.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:33:05.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:33:05.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:33:05.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-06-11T18:33:05.882 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-11T18:33:06.274 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' does not exist 2024-06-11T18:33:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-11T18:33:06.292 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-11T18:33:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'rbd' created 2024-06-11T18:33:06.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:33:07.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-11T18:33:10.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-11T18:33:10.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:33:10.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:33:10.856 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:33:10.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:33:10.858 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:33:10.858 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:33:10.858 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:33:10.859 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:33:10.859 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:33:11.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:33:11.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:33:11.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:33:11.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:33:11.002 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:33:11.002 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:33:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:33:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:33:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:33:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:11.337 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:33:11.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-06-11T18:33:11.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-06-11T18:33:11.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-06-11T18:33:11.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:11.564 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:33:11.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672972 2024-06-11T18:33:11.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672972 2024-06-11T18:33:11.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2024-06-11T18:33:11.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:11.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:11.760 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-06-11T18:33:11.762 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:33:11.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-06-11T18:33:11.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:11.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-06-11T18:33:11.764 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836494 2024-06-11T18:33:11.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-11T18:33:11.765 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:12.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836494 2024-06-11T18:33:12.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:13.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:33:13.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:13.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836494 2024-06-11T18:33:13.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:14.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:33:14.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:14.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-11T18:33:14.823 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:14.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:14.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672972 2024-06-11T18:33:14.826 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:33:14.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672972 2024-06-11T18:33:14.827 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:14.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672972 2024-06-11T18:33:14.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-11T18:33:14.830 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672972 2024-06-11T18:33:14.830 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:33:15.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-11T18:33:15.174 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:33:15.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:15.175 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-11T18:33:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:33:15.570 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:33:15.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-11T18:33:15.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:33:15.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:15.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:16.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-11T18:33:16.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:33:16.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:33:16.301 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-06-11T18:33:16.302 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-06-11T18:33:16.816 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'psr_pool' created 2024-06-11T18:33:16.840 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:33:17.842 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-11T18:33:17.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:33:17.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:33:17.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:33:17.843 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:33:17.844 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:33:17.844 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:33:17.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:33:17.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:33:17.845 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:33:17.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:33:17.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:33:17.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:33:17.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:33:17.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:33:17.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:33:18.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:33:18.428 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:33:18.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:33:18.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:18.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:33:18.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836497 2024-06-11T18:33:18.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836497 2024-06-11T18:33:18.583 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836497' 2024-06-11T18:33:18.584 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:18.584 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:33:18.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-06-11T18:33:18.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-06-11T18:33:18.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2024-06-11T18:33:18.728 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:18.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836497 2024-06-11T18:33:18.729 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:18.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:33:18.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836497 2024-06-11T18:33:18.731 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:18.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836497 2024-06-11T18:33:18.732 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836497 2024-06-11T18:33:18.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-11T18:33:18.733 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:19.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836495 -lt 21474836497 2024-06-11T18:33:19.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:20.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:33:20.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:20.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-11T18:33:20.404 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:20.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-06-11T18:33:20.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:20.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:33:20.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-06-11T18:33:20.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:20.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-06-11T18:33:20.410 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672974 2024-06-11T18:33:20.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-11T18:33:20.410 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:33:20.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-11T18:33:20.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:33:20.792 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:20.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:21.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:33:21.177 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:33:21.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:33:21.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:33:21.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:33:21.178 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:33:21.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:33:21.179 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:33:21.545 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:33:21.546 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:33:21.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:21.547 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-06-11T18:33:21.932 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-06-11T18:33:21.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-06-11T18:33:22.419 INFO:tasks.workunit.client.0.smithi012.stderr:noscrub is unset 2024-06-11T18:33:22.444 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-06-11T18:33:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is unset 2024-06-11T18:33:22.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-11T18:33:22.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-11T18:33:22.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-06-11T18:33:23.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-06-11T18:33:23.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-06-11T18:33:23.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-06-11T18:33:23.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-06-11T18:33:23.346 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:33:23.752 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-06-11T18:33:23.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-06-11T18:33:23.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-06-11T18:33:23.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-06-11T18:33:23.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-06-11T18:33:23.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:33:24.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-11T18:33:24.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-11T18:33:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-11T18:33:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-11T18:33:24.096 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:33:24.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:33:24.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:33:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:33:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:33:24.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:33:25.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:33:25.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:33:25.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:33:25.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:33:25.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:33:25.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:33:25.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:33:25.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-11T18:33:27.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:33:27.614 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:33:27.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:33:27.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:33:27.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:33:27.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:33:27.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:33:27.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:start osd.0 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-11T18:33:27.621 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:33:27.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:33:27.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:33:27.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:33:27.622 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:33:27.657 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:33:27.651+0000 7f13348c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:33:27.697 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:33:27.691+0000 7f13348c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:33:27.699 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:33:27.695+0000 7f13348c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:0 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:27.954 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:33:28.233 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:33:28.227+0000 7f13348c47c0 -1 Falling back to public interface 2024-06-11T18:33:28.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:29.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:29.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:29.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:33:29.338 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:33:29.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:29.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:33:29.674 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:33:30.676 INFO:tasks.workunit.client.0.smithi012.stderr:2 2024-06-11T18:33:30.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:30.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:33:31.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:33:31.133 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:33:31.119+0000 7f13348c47c0 -1 osd.0 25 log_to_monitors true 2024-06-11T18:33:32.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:33:32.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:33:32.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:33:32.015 INFO:tasks.workunit.client.0.smithi012.stderr:3 2024-06-11T18:33:32.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:33:32.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:osd.0 up in weight 1 up_from 29 up_thru 21 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/4046430781,v1:127.0.0.1:6803/4046430781] [v2:127.0.0.1:6804/4046430781,v1:127.0.0.1:6805/4046430781] exists,up ff2b5afa-537a-44fb-9ab9-8e67129672f2 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:33:32.372 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:33:32.373 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:33:32.374 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:33:32.374 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:33:32.374 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:33:32.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:33:32.375 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:33:32.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:33:32.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:33:32.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:33:32.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:33:32.501 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:33:32.501 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:33:33.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:33:33.210 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:33:33.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:33:33.210 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:33.211 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:33:33.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051586 2024-06-11T18:33:33.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051586 2024-06-11T18:33:33.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051586' 2024-06-11T18:33:33.470 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:33.471 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:33:33.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-06-11T18:33:33.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-06-11T18:33:33.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051586 1-42949672978' 2024-06-11T18:33:33.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:33.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051586 2024-06-11T18:33:33.780 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:33.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:33:33.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051586 2024-06-11T18:33:33.782 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:33.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051586 2024-06-11T18:33:33.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-06-11T18:33:33.784 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.0 seq 124554051586 2024-06-11T18:33:33.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:34.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051586 2024-06-11T18:33:34.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:35.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:33:35.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:35.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051586 2024-06-11T18:33:35.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:36.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-11T18:33:36.474 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:36.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051586 -lt 124554051586 2024-06-11T18:33:36.804 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:36.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-06-11T18:33:36.805 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:36.807 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:33:36.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:36.808 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-06-11T18:33:36.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-06-11T18:33:36.810 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-11T18:33:36.810 INFO:tasks.workunit.client.0.smithi012.stderr:waiting osd.1 seq 42949672978 2024-06-11T18:33:36.810 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:33:37.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-11T18:33:37.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:33:37.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:37.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:37.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-11T18:33:37.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:33:37.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:33:37.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:33:37.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:33:37.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:33:37.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:33:37.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:33:37.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-11T18:33:37.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:33:37.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:33:37.936 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:33:38.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-11T18:33:38.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:33:38.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:33:38.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-06-11T18:33:38.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-11T18:33:38.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-06-11T18:33:38.364 INFO:tasks.workunit.client.0.smithi012.stderr:No scrub information available for pg 2.0 2024-06-11T18:33:38.364 INFO:tasks.workunit.client.0.smithi012.stderr:error 2: (2) No such file or directory 2024-06-11T18:33:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-06-11T18:33:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-06-11T18:33:38.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: local pgid=2.0 2024-06-11T18:33:38.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:38.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:38.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:38.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:38.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: pg_deep_scrub: local last_scrub=2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:38.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-06-11T18:33:38.995 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-06-11T18:33:39.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: pg_deep_scrub: wait_for_scrub 2.0 2024-06-11T18:33:16.727621+0000 last_deep_scrub_stamp 2024-06-11T18:33:39.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:33:39.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:39.011 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-11T18:33:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:33:39.012 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:39.012 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:39.012 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:39.012 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:39.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:39.013 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:39.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:16.727621+0000 '>' 2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:39.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:40.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:40.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:40.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:40.325 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:40.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:40.326 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:16.727621+0000 '>' 2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:41.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:41.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:41.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:41.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:41.652 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:41.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:41.653 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:41.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:16.727621+0000 '>' 2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:41.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:42.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:42.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:42.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:42.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:42.990 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:42.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:42.991 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:43.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:16.727621+0000 '>' 2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:43.318 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:44.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-11T18:33:44.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:44.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-11T18:33:44.321 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:44.322 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-11T18:33:44.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:39.184058+0000 '>' 2024-06-11T18:33:16.727621+0000 2024-06-11T18:33:44.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:33:44.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-11T18:33:44.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-11T18:33:44.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-06-11T18:33:44.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-11T18:33:44.688 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:33:44.688 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:33:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:33:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:33:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:33:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:45.005 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:33:45.188 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051589 2024-06-11T18:33:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051589 2024-06-11T18:33:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051589' 2024-06-11T18:33:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:45.189 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:33:45.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-06-11T18:33:45.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-06-11T18:33:45.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051589 1-42949672981' 2024-06-11T18:33:45.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:45.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051589 2024-06-11T18:33:45.328 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:45.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:33:45.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051589 2024-06-11T18:33:45.330 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:45.331 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051589 2024-06-11T18:33:45.331 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 124554051589 2024-06-11T18:33:45.332 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-06-11T18:33:45.332 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:45.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051589 -lt 124554051589 2024-06-11T18:33:45.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:45.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-06-11T18:33:45.664 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:45.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:33:45.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:45.668 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-06-11T18:33:45.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-06-11T18:33:45.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-11T18:33:45.670 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672981 2024-06-11T18:33:45.670 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:33:45.978 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-11T18:33:45.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-06-11T18:33:45.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:45.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:45.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:45.979 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-11T18:33:46.441 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:33:46.441 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": false, 2024-06-11T18:33:46.442 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T18:33:46.442 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-06-04T18:32:06.441731+0000" 2024-06-11T18:33:46.442 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:33:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:33:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:46.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:33:46.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:33:46.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:46.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:46.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:46.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:46.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:46.457 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:46.759 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:39.184058+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:46.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:47.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:47.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:47.762 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:47.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:47.763 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:48.087 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:06.441731+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:48.088 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:49.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:49.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:49.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:49.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:49.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:49.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:49.091 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:49.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:06.441731+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:49.415 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:50.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:50.416 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:50.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:50.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:50.418 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:50.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:50.419 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:50.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:06.441731+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:50.750 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:51.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:51.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:51.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:51.752 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:51.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:51.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:51.753 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:52.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:06.441731+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:52.097 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:53.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:53.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:53.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:53.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:53.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:53.100 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:53.101 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:53.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:06.441731+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:53.426 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:33:54.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:33:54.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:33:54.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:33:54.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:33:54.428 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:33:54.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:33:54.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:33:54.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:33:47.273872+0000 '>' 2024-06-11T18:33:39.184058+0000 2024-06-11T18:33:54.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:33:54.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-06-11T18:33:54.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-11T18:33:54.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-06-11T18:33:54.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-11T18:33:54.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-06-11T18:33:55.235 INFO:tasks.workunit.client.0.smithi012.stderr:nodeep-scrub is set 2024-06-11T18:33:55.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-06-11T18:33:55.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-11T18:33:55.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:33:55.413 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:33:55.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:33:55.815 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:33:55.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:33:55.816 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:55.816 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:33:56.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051592 2024-06-11T18:33:56.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051592 2024-06-11T18:33:56.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051592' 2024-06-11T18:33:56.001 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:33:56.001 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:33:56.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-11T18:33:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-11T18:33:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051592 1-42949672985' 2024-06-11T18:33:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:56.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051592 2024-06-11T18:33:56.146 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:56.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:33:56.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051592 2024-06-11T18:33:56.148 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:56.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051592 2024-06-11T18:33:56.150 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 124554051592 2024-06-11T18:33:56.150 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-06-11T18:33:56.150 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:56.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051591 -lt 124554051592 2024-06-11T18:33:56.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:33:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:33:57.481 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:33:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051593 -lt 124554051592 2024-06-11T18:33:57.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-11T18:33:57.927 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672985 2024-06-11T18:33:57.928 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:33:58.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-06-11T18:33:58.238 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-06-11T18:34:03.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-11T18:34:03.388 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:34:03.388 INFO:tasks.workunit.client.0.smithi012.stdout: "deep": false, 2024-06-11T18:34:03.389 INFO:tasks.workunit.client.0.smithi012.stdout: "must": false, 2024-06-11T18:34:03.389 INFO:tasks.workunit.client.0.smithi012.stdout: "stamp": "2024-06-04T18:32:23.388734+0000" 2024-06-11T18:34:03.389 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:34:03.405 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-06-11T18:34:03.406 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-06-11T18:34:03.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-06-11T18:34:03.407 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-06-11T18:34:04.409 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-06-11T18:34:04.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-06-11T18:34:04.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-06-11T18:34:04.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-06-11T18:34:04.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-11T18:34:04.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-06-11T18:34:04.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-11T18:34:04.455 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-11T18:34:04.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:34:04.456 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:34:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:34:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:34:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:34:04.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:34:04.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:34:04.981 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051595 2024-06-11T18:34:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051595 2024-06-11T18:34:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051595' 2024-06-11T18:34:04.982 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:34:04.982 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:34:05.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672987 2024-06-11T18:34:05.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672987 2024-06-11T18:34:05.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051595 1-42949672987' 2024-06-11T18:34:05.133 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:34:05.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051595 2024-06-11T18:34:05.133 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:34:05.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:34:05.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051595 2024-06-11T18:34:05.135 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:34:05.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051595 2024-06-11T18:34:05.137 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051595' 2024-06-11T18:34:05.137 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 124554051595 2024-06-11T18:34:05.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:34:05.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051594 -lt 124554051595 2024-06-11T18:34:05.466 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:34:06.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:34:06.469 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:34:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051595 -lt 124554051595 2024-06-11T18:34:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:34:06.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672987 2024-06-11T18:34:06.802 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:34:06.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:34:06.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672987 2024-06-11T18:34:06.804 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:34:06.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672987 2024-06-11T18:34:06.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-06-11T18:34:06.806 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672987 2024-06-11T18:34:06.806 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:34:07.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672987 -lt 42949672987 2024-06-11T18:34:07.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-06-11T18:34:07.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: pg_schedule_scrub: local pgid=2.0 2024-06-11T18:34:07.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-06-11T18:34:07.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:07.137 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:07.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:07.138 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:07.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: local last_scrub=2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:07.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_schedule_scrub: ceph pg scrub 2.0 2024-06-11T18:34:07.778 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 2.0 on osd.0 to scrub 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_schedule_scrub: wait_for_scrub 2.0 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=2.0 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:34:07.796 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:07.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:07.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:07.797 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:07.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:07.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:23.388734+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:34:09.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:34:09.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:09.120 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:09.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:09.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:09.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:09.121 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:09.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:23.388734+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:09.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:34:10.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:34:10.458 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:10.459 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:10.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:10.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:10.460 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:10.461 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:10.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:23.388734+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:10.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:34:11.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:34:11.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:11.794 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:11.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:11.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:11.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:11.795 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:12.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:23.388734+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:12.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:34:13.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:34:13.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:13.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:13.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:13.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:13.131 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:13.132 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:13.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-04T18:32:23.388734+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:13.461 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:34:14.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:34:14.463 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:34:14.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-11T18:34:14.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-11T18:34:14.464 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:34:14.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:34:14.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-11T18:34:14.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:34:08.373154+0000 '>' 2024-06-04T18:32:23.388734+0000 2024-06-11T18:34:14.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:34:14.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-06-11T18:34:14.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-11T18:34:14.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-06-11T18:34:14.795 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-06-11T18:34:14.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:34:14.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:34:14.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:34:14.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:34:14.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:34:14.828 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:34:14.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:34:14.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:34:14.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:34:14.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:34:14.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:34:14.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:34:14.939 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:34:14.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:34:14.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:34:14.940 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:34:14.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:34:14.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:34:14.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:34:14.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:34:14.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:34:14.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:34:14.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:34:14.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:34:14.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:14.958 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:14.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:34:14.960 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:34:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:34:14.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:34:14.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:34:14.962 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:34:14.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:34:14.963 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:34:14.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:34:14.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:34:14.965 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:34:14.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:34:14.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:34:14.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:34:14.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:34:14.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:34:14.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:34:14.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:14.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:14.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:34:14.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-06-11T18:34:14.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:34:14.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:34:14.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:34:14.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:34:14.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:34:14.976 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:34:15.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:34:15.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:34:15.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:34:15.527 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:34:15.527 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:15.528 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:15.528 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:34:15.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:34:15.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:34:15.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:34:15.778 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:34:15.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:34:15.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:34:15.779 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:34:15.779 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:34:15.780 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:34:15.780 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:34:15.780 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:34:15.780 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:15.780 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:15.781 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:34:15.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:34:15.781 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:34:15.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:34:15.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:34:15.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:34:15.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:34:15.897 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:34:15.898 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:34:15.898 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:34:15.898 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:34:15.899 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:34:15.899 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:15.899 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:15.899 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:34:15.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:34:15.900 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:34:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:34:16.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:34:16.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:34:16.923 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:34:16.924 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:34:16.924 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:16.924 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:16.924 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:34:16.925 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:34:16.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:34:16.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:34:16.963 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-06-11T18:34:16.964 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:34:16.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:34:16.968 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:34:16.969 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:34:16.969 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:34:16.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:34:16.970 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:34:16.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:34:16.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:34:16.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3cf19fc8-f196-4f2b-a332-374871fa99f6 2024-06-11T18:34:16.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3cf19fc8-f196-4f2b-a332-374871fa99f6' 2024-06-11T18:34:16.972 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 3cf19fc8-f196-4f2b-a332-374871fa99f6 2024-06-11T18:34:16.972 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:34:16.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQComGhmoKbyOhAATPftfNxjZD4gLEiPeazunw== 2024-06-11T18:34:16.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQComGhmoKbyOhAATPftfNxjZD4gLEiPeazunw=="}' 2024-06-11T18:34:16.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3cf19fc8-f196-4f2b-a332-374871fa99f6 -i td/osd-scrub-repair/0/new.json 2024-06-11T18:34:17.282 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:34:17.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:34:17.299 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQComGhmoKbyOhAATPftfNxjZD4gLEiPeazunw== --osd-uuid 3cf19fc8-f196-4f2b-a332-374871fa99f6 2024-06-11T18:34:17.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.327+0000 7fa92f8997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:17.366 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.359+0000 7fa92f8997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:17.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.363+0000 7fa92f8997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:17.367 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.363+0000 7fa92f8997c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:34:17.368 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.363+0000 7fa92f8997c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:34:17.368 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.363+0000 7fa92f8997c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:34:17.458 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:17.451+0000 7fa92f8997c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:34:25.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:34:25.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:34:25.203 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:34:25.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:34:25.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:34:25.612 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:34:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:34:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:34:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:34:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:34:25.613 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:34:25.647 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:25.643+0000 7fa23710e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:25.680 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:25.675+0000 7fa23710e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:25.681 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:25.675+0000 7fa23710e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:34:25.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:34:25.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:34:25.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:34:25.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:25.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:26.237 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:26.231+0000 7fa23710e7c0 -1 Falling back to public interface 2024-06-11T18:34:26.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:27.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:27.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:27.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:34:27.365 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:34:27.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:27.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:27.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:28.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:28.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:28.703 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:34:28.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:34:28.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:28.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:29.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:29.055 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:34:29.050+0000 7fa23710e7c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:34:30.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:30.046 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:30.046 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:34:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:34:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:30.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:31.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:31.429 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:31.430 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:34:31.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:34:31.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:31.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:31.765 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:32.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:32.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:32.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:34:32.767 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:34:32.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:32.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:33.222 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:34.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:34.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:34.224 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:34:34.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:34:34.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:34.225 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:34.563 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:35.564 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:35.565 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:34:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:34:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:35.565 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:35.921 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:36.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:36.923 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:36.923 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:34:36.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:34:36.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:36.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:37.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:34:38.342 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:34:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:38.673 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:39.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:39.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:39.675 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:34:39.675 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:34:39.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:39.676 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:41.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:41.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:41.148 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:34:41.149 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:34:41.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:41.149 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:42.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:42.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:42.476 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:34:42.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:34:42.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:42.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:42.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:43.938 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:43.939 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:34:43.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:34:43.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:43.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:44.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:45.435 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:45.436 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:34:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:34:45.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:45.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:45.774 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:46.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:46.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:46.776 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:34:46.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:34:46.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:46.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:47.201 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:48.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:48.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:48.204 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:34:48.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:34:48.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:48.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:49.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:50.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:50.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:50.229 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:34:50.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:34:50.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:50.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:50.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:51.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:51.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:51.979 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:34:51.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:34:51.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:51.980 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:52.457 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:53.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:53.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:53.460 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:34:53.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:34:53.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:53.460 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:54.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:55.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:55.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:55.229 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:34:55.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:34:55.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:55.229 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:55.939 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:56.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:56.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:56.941 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:34:56.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:34:56.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:56.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:34:58.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:34:58.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:34:58.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:34:58.413 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:34:58.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:34:58.414 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:34:59.180 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:00.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:00.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:00.182 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:35:00.183 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:35:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:35:00.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:01.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:01.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:01.849 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:35:01.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:35:01.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:01.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:35:02.200 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3189052933,v1:127.0.0.1:6803/3189052933] [v2:127.0.0.1:6804/3189052933,v1:127.0.0.1:6805/3189052933] exists,up 3cf19fc8-f196-4f2b-a332-374871fa99f6 2024-06-11T18:35:02.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:35:02.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:35:02.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:35:02.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:35:02.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:35:02.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:35:02.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:35:02.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:35:02.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:35:02.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:35:02.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:35:02.206 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:35:02.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=489ccba5-409a-4970-b3a0-cc18cac5647e 2024-06-11T18:35:02.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 489ccba5-409a-4970-b3a0-cc18cac5647e' 2024-06-11T18:35:02.208 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 489ccba5-409a-4970-b3a0-cc18cac5647e 2024-06-11T18:35:02.208 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:35:02.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWmGhmDjSnDRAALFtsDrxh+B6WkFPNQwX89Q== 2024-06-11T18:35:02.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWmGhmDjSnDRAALFtsDrxh+B6WkFPNQwX89Q=="}' 2024-06-11T18:35:02.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 489ccba5-409a-4970-b3a0-cc18cac5647e -i td/osd-scrub-repair/1/new.json 2024-06-11T18:35:02.742 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:35:02.763 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:35:02.764 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDWmGhmDjSnDRAALFtsDrxh+B6WkFPNQwX89Q== --osd-uuid 489ccba5-409a-4970-b3a0-cc18cac5647e 2024-06-11T18:35:02.804 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.802+0000 7fa417d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:02.877 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.874+0000 7fa417d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:02.880 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.878+0000 7fa417d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:02.880 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.878+0000 7fa417d897c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:35:02.880 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.878+0000 7fa417d897c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:35:02.880 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.878+0000 7fa417d897c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:35:02.975 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:02.974+0000 7fa417d897c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:35:10.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:35:10.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:35:10.989 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:35:10.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:35:10.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:35:11.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:35:11.395 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:35:11.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:35:11.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:35:11.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:35:11.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:35:11.430 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:11.430+0000 7f0aac7337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:11.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:11.470+0000 7f0aac7337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:11.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:11.474+0000 7f0aac7337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:35:11.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:35:11.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:35:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:35:11.720 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:12.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:12.026+0000 7f0aac7337c0 -1 Falling back to public interface 2024-06-11T18:35:12.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:13.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:13.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:13.094 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:35:13.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:35:13.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:13.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:13.421 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:35:14.423 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:35:14.423 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:14.424 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:14.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:14.992 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:35:14.990+0000 7f0aac7337c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:35:15.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:15.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:15.755 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:35:15.755 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:35:15.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:15.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:16.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:17.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:17.242 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:17.242 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:35:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:35:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:17.243 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:17.922 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:18.924 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:18.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:18.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:35:18.925 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:35:18.925 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:18.926 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:19.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:20.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:20.388 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:20.388 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:35:20.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:35:20.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:20.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:20.818 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:21.819 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:21.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:21.820 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:35:21.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:35:21.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:21.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:22.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:23.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:23.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:23.259 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:35:23.259 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:35:23.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:23.260 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:23.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:24.760 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:24.760 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:35:24.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:35:24.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:24.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:25.173 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:26.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:26.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:26.176 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:35:26.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:35:26.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:26.176 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:26.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:27.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:27.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:27.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:35:27.528 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:35:27.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:27.529 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:27.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:28.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:28.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:28.890 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:35:28.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:35:28.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:28.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:29.231 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:30.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:30.233 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:30.233 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:35:30.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:35:30.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:30.234 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:30.707 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:31.709 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:31.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:31.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:35:31.710 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:35:31.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:31.710 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:32.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:33.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:33.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:33.312 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:35:33.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:35:33.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:33.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:33.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:34.657 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:34.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:34.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:35:34.658 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:35:34.658 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:34.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:35.163 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:36.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:36.165 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:36.166 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:35:36.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:35:36.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:36.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:37.192 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:38.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:38.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:38.194 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:35:38.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:35:38.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:38.195 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:38.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:39.726 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:35:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:35:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:40.349 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:41.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:41.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:41.351 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:35:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:35:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:42.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:43.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:43.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:43.138 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:35:43.138 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:35:43.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:43.139 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:43.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:44.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:44.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:44.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:35:44.944 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:35:44.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:44.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:46.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:46.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:46.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:35:46.382 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:35:46.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:46.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:47.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:35:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:35:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:35:48.135 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:35:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:35:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:35:48.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:35:48.476 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3650163411,v1:127.0.0.1:6811/3650163411] [v2:127.0.0.1:6812/3650163411,v1:127.0.0.1:6813/3650163411] exists,up 489ccba5-409a-4970-b3a0-cc18cac5647e 2024-06-11T18:35:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:35:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:35:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:35:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-06-11T18:35:48.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-11T18:35:48.935 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T18:35:48.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:35:49.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-06-11T18:35:50.859 INFO:tasks.workunit.client.0.smithi012.stderr:set pool 1 size to 2 2024-06-11T18:35:50.886 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-06-11T18:35:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:35:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:35:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:35:50.887 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:35:50.888 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:35:50.888 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:35:50.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:35:50.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:35:50.889 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:35:51.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:35:51.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:35:51.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:35:51.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:35:51.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:35:51.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:35:51.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:35:51.630 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:35:51.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:35:51.630 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:35:51.630 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:35:51.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836497 2024-06-11T18:35:51.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836497 2024-06-11T18:35:51.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836497' 2024-06-11T18:35:51.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:35:51.847 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:35:51.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-06-11T18:35:51.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-06-11T18:35:51.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2024-06-11T18:35:51.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:35:51.994 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836497 2024-06-11T18:35:51.995 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:35:51.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:35:51.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836497 2024-06-11T18:35:51.996 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:35:51.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836497 2024-06-11T18:35:51.998 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-11T18:35:51.998 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836497 2024-06-11T18:35:51.998 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:35:52.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-11T18:35:52.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:35:52.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-06-11T18:35:52.488 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:35:52.490 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:35:52.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-06-11T18:35:52.494 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:35:52.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-06-11T18:35:52.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-11T18:35:52.494 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672974 2024-06-11T18:35:52.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:35:52.820 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672974 2024-06-11T18:35:52.821 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:35:53.822 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:35:53.823 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:35:54.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672974 2024-06-11T18:35:54.160 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:35:54.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:35:54.161 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:35:54.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:35:54.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:35:54.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:35:54.588 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:35:54.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:35:54.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:35:54.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:35:54.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:35:54.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:35:54.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:35:54.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:35:54.960 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-06-11T18:35:55.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-06-11T18:35:55.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:55.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:55.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:55.601 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:55.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:55.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.018 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.412 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.604 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:56.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:56.996 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:57.189 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:57.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:57.389 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:58.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:58.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:58.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:58.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:58.418 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:58.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:58.803 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:59.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:59.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:59.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:59.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:59.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:59.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:35:59.805 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:35:59.806 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:00.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:00.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:00.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:00.258 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:00.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:00.459 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:00.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:00.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:00.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:00.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:01.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-11T18:36:01.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:36:01.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-06-11T18:36:01.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T18:36:01.422 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-11T18:36:01.423 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-11T18:36:01.423 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T18:36:01.423 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T18:36:01.424 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T18:36:01.424 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:36:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T18:36:01.758 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-11T18:36:01.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-11T18:36:02.129 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:36:02.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-06-11T18:36:02.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-06-11T18:36:02.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T18:36:02.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T18:36:02.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T18:36:02.524 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:36:02.891 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-06-11T18:36:02.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:36:02.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:36:02.892 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:36:02.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:36:02.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:36:02.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:36:04.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:36:04.205 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:36:04.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:36:04.207 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:36:04.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:36:04.207 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:36:04.208 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:36:04.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:36:04.521 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-06-11T18:36:04.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:04.523 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-06-11T18:36:04.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:04.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-11T18:36:04.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:04.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:04.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:04.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:04.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:04.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-11T18:36:06.348 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-11T18:36:06.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:06.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-06-11T18:36:06.895 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:06.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-11T18:36:08.664 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:104778fc:::obj2:head# 2024-06-11T18:36:09.212 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:09.212 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-06-11T18:36:09.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:09.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-11T18:36:09.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:09.215 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:09.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:09.216 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:09.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:09.217 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-11T18:36:10.489 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:8dd16f86:::obj3:head# 2024-06-11T18:36:11.023 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:11.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-11T18:36:12.358 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:0ee9ae15:::obj4:head# 2024-06-11T18:36:12.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:12.894 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-11T18:36:13.819 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:c52c9666:::obj5:head# 2024-06-11T18:36:14.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:14.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:14.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-11T18:36:15.289 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:233a42c1:::obj6:head# 2024-06-11T18:36:15.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:15.825 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:15.827 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:15.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-11T18:36:16.766 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:9a09113a:::obj7:head# 2024-06-11T18:36:17.300 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:17.301 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:17.304 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-11T18:36:18.367 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:a557efb1:::obj8:head# 2024-06-11T18:36:18.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:18.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:18.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-11T18:36:19.837 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:fe351e27:::obj9:head# 2024-06-11T18:36:20.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:20.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:20.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-11T18:36:21.289 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:d1337354:::obj10:head# 2024-06-11T18:36:21.825 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:21.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-06-11T18:36:21.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:21.829 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-11T18:36:22.749 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:eeae2a94:::obj11:head# 2024-06-11T18:36:23.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:23.786 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-11T18:36:24.762 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:af5f95cb:::obj12:head# 2024-06-11T18:36:25.797 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:25.798 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-06-11T18:36:25.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:25.799 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:25.800 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-11T18:36:26.665 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:a61ad63e:::obj13:head# 2024-06-11T18:36:27.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:27.201 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:27.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-11T18:36:28.081 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:e0a44829:::obj14:head# 2024-06-11T18:36:28.617 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:28.618 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:28.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-11T18:36:29.484 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:ab946124:::obj15:head# 2024-06-11T18:36:30.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:30.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:30.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:30.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-11T18:36:30.913 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:461f8b5e:::obj16:head# 2024-06-11T18:36:31.449 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:31.516 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-06-11T18:36:31.516 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:31.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-11T18:36:32.481 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:24fc6e92:::obj17:head# 2024-06-11T18:36:33.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:33.018 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:33.020 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-11T18:36:33.890 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:b9836a99:::obj18:head# 2024-06-11T18:36:34.425 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:34.425 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-06-11T18:36:34.427 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:34.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-11T18:36:35.326 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:00b31b6c:::obj19:head# 2024-06-11T18:36:35.860 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-11T18:36:35.862 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:35.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-11T18:36:36.703 INFO:tasks.workunit.client.0.smithi012.stdout:remove #1:24b5611b:::obj20:head# 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:36:37.249 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:36:37.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:36:37.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:36:37.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:36:37.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:36:37.251 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:36:37.251 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:36:37.252 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:36:37.252 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:36:37.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:36:37.254 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:36:37.254 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:36:37.255 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:36:37.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:36:37.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:36:37.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:36:37.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:36:37.291 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:37.290+0000 7f76da9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:37.294 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:37.294+0000 7f76da9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:37.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:37.294+0000 7f76da9b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:37.586 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:36:37.587 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:36:37.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:37.587 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:36:37.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:37.834+0000 7f76da9b17c0 -1 Falling back to public interface 2024-06-11T18:36:37.966 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:38.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:38.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:38.968 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:36:38.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:36:38.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:38.969 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:36:39.294 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:40.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:40.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:40.296 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:36:40.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:36:40.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:40.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:36:40.631 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:40.629+0000 7f76da9b17c0 -1 osd.1 18 log_to_monitors true 2024-06-11T18:36:40.633 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:41.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:41.635 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:41.636 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:36:41.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:36:41.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:41.636 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:36:41.970 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/1032184253,v1:127.0.0.1:6803/1032184253] [v2:127.0.0.1:6804/1032184253,v1:127.0.0.1:6805/1032184253] exists,up 489ccba5-409a-4970-b3a0-cc18cac5647e 2024-06-11T18:36:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:36:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:36:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:36:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:36:41.974 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:36:41.975 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:36:41.976 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:36:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:36:42.008 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:42.005+0000 7fe4916407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:42.018 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:42.017+0000 7fe4916407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:42.017+0000 7fe4916407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:36:42.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:36:42.313 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:36:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:36:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:36:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:36:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:42.314 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:36:42.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:36:42.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:42.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:36:42.572 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:42.569+0000 7fe4916407c0 -1 Falling back to public interface 2024-06-11T18:36:42.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:43.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:43.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:43.667 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:36:43.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:36:43.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:43.668 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:36:44.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:36:45.041 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:36:45.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:45.042 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:36:45.365 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:36:45.361+0000 7fe4916407c0 -1 osd.0 16 log_to_monitors true 2024-06-11T18:36:45.376 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:36:46.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:36:46.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:36:46.378 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:36:46.378 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:36:46.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:36:46.379 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:36:46.782 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/1040970278,v1:127.0.0.1:6811/1040970278] [v2:127.0.0.1:6812/1040970278,v1:127.0.0.1:6813/1040970278] exists,up 3cf19fc8-f196-4f2b-a332-374871fa99f6 2024-06-11T18:36:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:36:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:36:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:36:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-06-11T18:36:46.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:36:46.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:36:46.784 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:36:46.784 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:36:46.785 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:36:46.785 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:36:46.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:36:46.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:36:46.785 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:36:46.911 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:36:47.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:36:47.257 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:36:47.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:36:47.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:36:47.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:36:47.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215106 2024-06-11T18:36:47.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215106 2024-06-11T18:36:47.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106' 2024-06-11T18:36:47.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:36:47.453 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:36:47.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313219 2024-06-11T18:36:47.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313219 2024-06-11T18:36:47.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215106 1-90194313219' 2024-06-11T18:36:47.605 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:36:47.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215106 2024-06-11T18:36:47.606 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:36:47.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:36:47.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215106 2024-06-11T18:36:47.608 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:36:47.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215106 2024-06-11T18:36:47.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-11T18:36:47.609 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 103079215106 2024-06-11T18:36:47.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:36:47.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 103079215106 2024-06-11T18:36:47.933 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:36:48.934 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:36:48.935 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:36:49.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-11T18:36:49.273 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:36:49.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-90194313219 2024-06-11T18:36:49.274 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:36:49.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:36:49.277 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-90194313219 2024-06-11T18:36:49.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:36:49.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313219 2024-06-11T18:36:49.279 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 90194313219 2024-06-11T18:36:49.279 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 90194313219' 2024-06-11T18:36:49.280 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:36:49.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313219 -lt 90194313219 2024-06-11T18:36:49.646 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:36:49.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:36:49.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:36:50.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:36:50.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:36:50.136 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:36:50.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:36:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:36:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:36:50.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:50.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:50.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:50.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:51.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:51.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-06-11T18:36:51.573 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-11T18:36:51.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:51.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T18:36:51.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:51.590 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:36:51.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:36:51.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:51.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:51.591 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:51.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:51.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:51.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:51.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:35:48.851086+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:51.917 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:36:52.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:36:52.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:52.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:52.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:52.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:52.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:52.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:53.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:35:48.851086+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:53.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:36:54.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:36:54.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:54.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:54.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:54.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:54.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:54.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:54.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:35:48.851086+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:54.589 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:36:55.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:36:55.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:55.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:55.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:55.592 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:55.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:55.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:55.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:35:48.851086+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:55.929 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:36:56.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:36:56.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:56.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:56.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:56.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:56.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:56.933 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:57.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:35:48.851086+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:57.255 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:36:58.256 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:36:58.257 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:36:58.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:36:58.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:36:58.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:36:58.258 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:36:58.259 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:36:58.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:36:51.723045+0000 '>' 2024-06-11T18:35:48.851086+0000 2024-06-11T18:36:58.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:36:58.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-06-11T18:36:58.592 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:36:58.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:36:58.593 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:36:58.593 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:36:58.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:36:58.594 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:36:58.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:36:58.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:36:58.594 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:36:58.727 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:36:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:36:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:36:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:36:59.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:36:59.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:36:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215109 2024-06-11T18:36:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215109 2024-06-11T18:36:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215109' 2024-06-11T18:36:59.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:36:59.204 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:36:59.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313222 2024-06-11T18:36:59.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313222 2024-06-11T18:36:59.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215109 1-90194313222' 2024-06-11T18:36:59.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:36:59.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215109 2024-06-11T18:36:59.349 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:36:59.351 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:36:59.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215109 2024-06-11T18:36:59.352 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:36:59.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215109 2024-06-11T18:36:59.354 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-06-11T18:36:59.354 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 103079215109 2024-06-11T18:36:59.354 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:36:59.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215108 -lt 103079215109 2024-06-11T18:36:59.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:37:00.661 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:37:00.662 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:37:00.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215109 -lt 103079215109 2024-06-11T18:37:00.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:37:01.000 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-90194313222 2024-06-11T18:37:01.001 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:37:01.003 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:37:01.004 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-90194313222 2024-06-11T18:37:01.005 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:37:01.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313222 2024-06-11T18:37:01.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 90194313222' 2024-06-11T18:37:01.007 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 90194313222 2024-06-11T18:37:01.007 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:37:01.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313222 -lt 90194313222 2024-06-11T18:37:01.368 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:37:01.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:37:01.369 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:37:01.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:37:01.811 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:37:01.812 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:37:02.193 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:37:02.222 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:37:02.223 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:37:02.223 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:37:02.594 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:37:02.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:37:02.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:37:02.595 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-06-11T18:37:02.927 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T18:37:02.927 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-11T18:36:53.189056+0000 16'30 25:123 [1,0] 1 [1,0] 1 16'30 2024-06-11T18:36:51.723045+0000 16'30 2024-06-11T18:36:51.723045+0000 0 1 periodic scrub scheduled @ 2024-06-12T18:36:51.723045+0000 20 0 2024-06-11T18:37:02.927 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T18:37:02.927 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T18:37:02.928 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T18:37:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-06-11T18:37:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:37:02.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:37:03.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:37:03.296 INFO:tasks.workunit.client.0.smithi012.stderr:1' 2024-06-11T18:37:03.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:37:03.296 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:37:03.296 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:37:03.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215111 2024-06-11T18:37:03.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215111 2024-06-11T18:37:03.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215111' 2024-06-11T18:37:03.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:37:03.477 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313224 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313224 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215111 1-90194313224' 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215111 2024-06-11T18:37:03.625 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:37:03.626 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:37:03.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215111 2024-06-11T18:37:03.627 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:37:03.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215111 2024-06-11T18:37:03.629 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 103079215111 2024-06-11T18:37:03.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2024-06-11T18:37:03.629 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:37:03.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215110 -lt 103079215111 2024-06-11T18:37:03.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:37:04.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:37:04.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:37:05.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215111 -lt 103079215111 2024-06-11T18:37:05.278 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:37:05.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-90194313224 2024-06-11T18:37:05.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:37:05.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:37:05.282 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-90194313224 2024-06-11T18:37:05.283 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:37:05.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313224 2024-06-11T18:37:05.284 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 90194313224' 2024-06-11T18:37:05.284 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 90194313224 2024-06-11T18:37:05.285 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:37:05.648 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313224 -lt 90194313224 2024-06-11T18:37:05.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-06-11T18:37:05.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-06-11T18:37:05.850 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:37:05.850 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes": 210, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects": 30, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_object_clones": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_object_copies": 60, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_missing_on_primary": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_missing": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_degraded": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_misplaced": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_unfound": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_dirty": 30, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_whiteouts": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_read": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_read_kb": 0, 2024-06-11T18:37:05.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_write": 30, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_write_kb": 30, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_scrub_errors": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shallow_scrub_errors": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_deep_scrub_errors": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_recovered": 20, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes_recovered": 140, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_keys_recovered": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_omap": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_hit_set_archive": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes_hit_set_archive": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_kb": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict": 0, 2024-06-11T18:37:05.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_kb": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_promote": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_mode_high": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_mode_low": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_mode_some": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_mode_full": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_pinned": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_legacy_snapsets": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_large_omap_objects": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_manifest": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_omap_bytes": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_omap_keys": 0, 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_repaired": 20 2024-06-11T18:37:05.853 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:37:05.854 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-06-11T18:37:05.854 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-06-11T18:37:06.005 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-06-11T18:37:06.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-06-11T18:37:06.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:06.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-06-11T18:37:06.300 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "osd": 1, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "up_from": 21, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "seq": 90194313224, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "num_pgs": 1, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "num_osds": 1, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_osds": 1, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_omap_osds": 1, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "kb": 104857600, 2024-06-11T18:37:06.316 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used": 28560, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_data": 328, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_omap": 4, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_meta": 28219, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_avail": 104829040, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "statfs": { 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "total": 107374182400, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "available": 107344936960, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "internally_reserved": 0, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "allocated": 335872, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "data_stored": 56243, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed": 0, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_allocated": 0, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_original": 0, 2024-06-11T18:37:06.317 INFO:tasks.workunit.client.0.smithi012.stdout: "omap_allocated": 4398, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "internal_metadata": 28896978 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "hb_peers": [ 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: 0 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "snap_trim_queue_len": 0, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "num_snap_trimming": 0, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shards_repaired": 10, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "op_queue_age_hist": { 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "histogram": [], 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "upper_bound": 1 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "perf_stat": { 2024-06-11T18:37:06.318 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ms": 0, 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ms": 0, 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ns": 0, 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ns": 0 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout: "alerts": [] 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:06.319 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-06-11T18:37:06.624 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:06.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-06-11T18:37:06.643 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-06-11T18:37:06.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-06-11T18:37:06.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:06.645 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-11T18:37:06.949 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "osd": 0, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "up_from": 24, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "seq": 103079215112, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "num_pgs": 1, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "num_osds": 1, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_osds": 1, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_omap_osds": 1, 2024-06-11T18:37:06.965 INFO:tasks.workunit.client.0.smithi012.stdout: "kb": 104857600, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used": 28560, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_data": 328, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_omap": 4, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_meta": 28219, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_avail": 104829040, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "statfs": { 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "total": 107374182400, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "available": 107344936960, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "internally_reserved": 0, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "allocated": 335872, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "data_stored": 56243, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed": 0, 2024-06-11T18:37:06.966 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_allocated": 0, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_original": 0, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "omap_allocated": 4401, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "internal_metadata": 28896975 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "hb_peers": [ 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: 1 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "snap_trim_queue_len": 0, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "num_snap_trimming": 0, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shards_repaired": 10, 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "op_queue_age_hist": { 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "histogram": [], 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: "upper_bound": 1 2024-06-11T18:37:06.967 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "perf_stat": { 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ms": 0, 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ms": 0, 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ns": 0, 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ns": 0 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout: "alerts": [] 2024-06-11T18:37:06.968 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:37:06.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:06.969 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-11T18:37:07.270 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:07.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-06-11T18:37:07.288 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-06-11T18:37:07.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-06-11T18:37:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:07.289 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-06-11T18:37:07.582 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "up_from": 0, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "seq": 0, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "num_pgs": 2, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "num_osds": 2, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_osds": 2, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_omap_osds": 2, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "kb": 209715200, 2024-06-11T18:37:07.598 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used": 57120, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_data": 656, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_omap": 8, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_meta": 56439, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_avail": 209658080, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "statfs": { 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "total": 214748364800, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "available": 214689873920, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "internally_reserved": 0, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "allocated": 671744, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "data_stored": 112486, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed": 0, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_allocated": 0, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_original": 0, 2024-06-11T18:37:07.599 INFO:tasks.workunit.client.0.smithi012.stdout: "omap_allocated": 8799, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "internal_metadata": 57793953 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "hb_peers": [], 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "snap_trim_queue_len": 0, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "num_snap_trimming": 0, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shards_repaired": 20, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "op_queue_age_hist": { 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "histogram": [], 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "upper_bound": 1 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "perf_stat": { 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ms": 0, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ms": 0, 2024-06-11T18:37:07.600 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ns": 0, 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ns": 0 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stdout: "alerts": [], 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stdout: "network_ping_times": [] 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-06-11T18:37:07.601 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-11T18:37:07.901 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:37:07.918 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:37:07.919 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:37:07.920 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:37:07.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:37:07.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:37:07.920 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:37:08.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:37:08.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:37:08.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:37:08.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:37:08.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:37:08.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:37:08.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:37:08.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:37:08.032 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:37:08.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:37:08.033 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:37:08.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:37:08.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:37:08.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:37:08.049 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:37:08.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:37:08.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:37:08.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:37:08.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-11T18:37:08.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:37:08.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:37:08.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:37:08.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:37:08.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:37:08.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:37:08.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:37:08.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:37:08.055 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:37:08.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:37:08.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:37:08.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:37:08.058 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:37:08.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:37:08.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-11T18:37:08.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:37:08.060 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:37:08.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.061 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:37:08.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:37:08.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:37:08.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-11T18:37:08.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-11T18:37:08.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.063 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61904 2024-06-11T18:37:08.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-11T18:37:08.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-11T18:37:08.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-11T18:37:08.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-11T18:37:08.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-06-11T18:37:08.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-11T18:37:08.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-11T18:37:08.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-11T18:37:08.577 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:37:08.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:37:08.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:37:08.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.578 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.578 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:08.579 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-11T18:37:08.855 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-11T18:37:08.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-11T18:37:08.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:37:08.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:37:08.857 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-11T18:37:08.858 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-11T18:37:08.858 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:37:08.858 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:37:08.858 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:37:08.859 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get fsid 2024-06-11T18:37:08.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-11T18:37:08.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-11T18:37:08.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-11T18:37:08.977 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-11T18:37:08.978 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:08.979 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61904/ceph-mon.a.asok 2024-06-11T18:37:08.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-11T18:37:08.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61904/ceph-mon.a.asok config get mon_host 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-11T18:37:09.098 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-11T18:37:09.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-11T18:37:10.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-11T18:37:10.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:37:10.023 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:37:10.024 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:37:10.024 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:10.024 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:10.024 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:10.025 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:37:10.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-11T18:37:10.061 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:37:10.062 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-06-11T18:37:10.064 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-06-11T18:37:10.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:37:10.066 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:37:10.070 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:37:10.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:37:10.074 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:37:10.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d31479b-a2ab-4e34-828b-2ddf511fbfbf 2024-06-11T18:37:10.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7d31479b-a2ab-4e34-828b-2ddf511fbfbf' 2024-06-11T18:37:10.074 INFO:tasks.workunit.client.0.smithi012.stdout:add osd0 7d31479b-a2ab-4e34-828b-2ddf511fbfbf 2024-06-11T18:37:10.075 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:37:10.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWmWhmiQI6BRAAY49haXgLL1JOB4ZrUZRCLA== 2024-06-11T18:37:10.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWmWhmiQI6BRAAY49haXgLL1JOB4ZrUZRCLA=="}' 2024-06-11T18:37:10.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d31479b-a2ab-4e34-828b-2ddf511fbfbf -i td/osd-scrub-repair/0/new.json 2024-06-11T18:37:10.391 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:37:10.408 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-11T18:37:10.410 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBWmWhmiQI6BRAAY49haXgLL1JOB4ZrUZRCLA== --osd-uuid 7d31479b-a2ab-4e34-828b-2ddf511fbfbf 2024-06-11T18:37:10.447 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.445+0000 7f2d842167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:10.484 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.481+0000 7f2d842167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.481+0000 7f2d842167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.485+0000 7f2d842167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:37:10.486 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.485+0000 7f2d842167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.485+0000 7f2d842167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-11T18:37:10.592 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:10.589+0000 7f2d842167c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-11T18:37:18.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-11T18:37:18.337 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:37:18.338 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-11T18:37:18.339 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd0 key to auth repository 2024-06-11T18:37:18.339 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:37:18.760 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:37:18.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-11T18:37:18.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:37:18.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:37:18.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:37:18.761 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:37:18.796 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:18.793+0000 7f100ea787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:18.841 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:18.837+0000 7f100ea787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:18.843 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:18.841+0000 7f100ea787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:19.077 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:37:19.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:37:19.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:19.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:19.396 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:19.393+0000 7f100ea787c0 -1 Falling back to public interface 2024-06-11T18:37:19.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:20.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:20.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:20.442 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:37:20.442 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:37:20.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:20.443 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:20.776 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:21.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:21.778 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:21.778 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:37:21.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:37:21.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:21.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:22.110 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:22.319 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:22.317+0000 7f100ea787c0 -1 osd.0 0 log_to_monitors true 2024-06-11T18:37:23.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:23.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:23.112 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:37:23.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:37:23.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:23.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:23.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:24.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:24.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:24.525 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:37:24.525 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:37:24.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:24.526 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:24.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:25.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:25.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:25.874 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:37:25.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:37:25.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:25.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:26.218 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:27.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:27.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:27.220 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:37:27.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:37:27.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:27.221 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:27.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:28.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:28.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:28.722 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:37:28.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:37:28.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:28.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:29.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:30.062 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:30.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:30.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:37:30.063 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:37:30.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:30.063 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:30.450 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:31.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:31.452 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:31.452 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:37:31.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:37:31.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:31.453 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:31.777 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:32.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:32.779 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:32.779 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:37:32.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:37:32.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:32.780 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:33.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:34.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:34.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:34.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:37:34.203 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:37:34.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:34.204 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:34.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:35.696 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:35.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:35.697 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:37:35.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:37:35.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:35.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:36.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:37.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:37.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:37.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:37:37.036 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:37:37.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:37.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:37.380 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:38.381 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:38.382 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:37:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:37:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:38.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:38.717 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:39.719 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:37:39.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:37:39.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:39.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:40.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:41.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:37:41.052 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:37:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:41.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:42.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:37:42.679 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:37:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:43.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:37:44.366 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:37:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:44.367 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:45.054 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:46.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:46.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:46.056 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:37:46.056 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:37:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:46.730 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:47.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:47.732 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:47.732 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:37:47.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:37:47.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:47.733 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:48.321 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:49.322 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:49.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:49.323 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:37:49.323 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:37:49.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:49.324 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:50.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:51.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:51.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:51.039 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:37:51.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:37:51.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:51.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:51.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:52.961 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:52.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:52.962 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:37:52.962 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:37:52.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:52.963 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:53.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:37:54.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:37:54.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:37:54.502 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:37:54.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:37:54.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:37:54.503 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:37:54.846 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/405075214,v1:127.0.0.1:6803/405075214] [v2:127.0.0.1:6804/405075214,v1:127.0.0.1:6805/405075214] exists,up 7d31479b-a2ab-4e34-828b-2ddf511fbfbf 2024-06-11T18:37:54.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:37:54.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:37:54.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:37:54.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:37:54.848 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:37:54.849 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:37:54.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:37:54.850 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:37:54.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:37:54.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:37:54.851 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:37:54.852 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:37:54.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:37:54.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:37:54.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:37:54.853 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:37:54.853 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:37:54.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30af0c36-d79a-4c73-aad5-b7733ed88642 2024-06-11T18:37:54.854 INFO:tasks.workunit.client.0.smithi012.stdout:add osd1 30af0c36-d79a-4c73-aad5-b7733ed88642 2024-06-11T18:37:54.854 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 30af0c36-d79a-4c73-aad5-b7733ed88642' 2024-06-11T18:37:54.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:37:54.872 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCmWhmQ5n5MxAAOFl1HI9d7rAT+TXCUSaTSA== 2024-06-11T18:37:54.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCmWhmQ5n5MxAAOFl1HI9d7rAT+TXCUSaTSA=="}' 2024-06-11T18:37:54.873 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30af0c36-d79a-4c73-aad5-b7733ed88642 -i td/osd-scrub-repair/1/new.json 2024-06-11T18:37:55.362 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:37:55.382 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-11T18:37:55.383 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCCmWhmQ5n5MxAAOFl1HI9d7rAT+TXCUSaTSA== --osd-uuid 30af0c36-d79a-4c73-aad5-b7733ed88642 2024-06-11T18:37:55.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.413+0000 7f8310d767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:55.507 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.505+0000 7f8310d767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:55.508 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.505+0000 7f8310d767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:37:55.509 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.505+0000 7f8310d767c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:37:55.509 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.505+0000 7f8310d767c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:37:55.509 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.505+0000 7f8310d767c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-11T18:37:55.593 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:37:55.589+0000 7f8310d767c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-11T18:38:03.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-11T18:38:03.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:38:03.475 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd1 key to auth repository 2024-06-11T18:38:03.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-11T18:38:03.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:38:03.882 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:38:03.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-11T18:38:03.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:38:03.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:38:03.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:38:03.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:38:03.918 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:03.913+0000 7fb2199927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:03.951 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:03.949+0000 7fb2199927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:03.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:03.949+0000 7fb2199927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:04.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:38:04.241 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:38:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:04.241 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:04.505 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:04.501+0000 7fb2199927c0 -1 Falling back to public interface 2024-06-11T18:38:04.618 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:05.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:05.621 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:38:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:38:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:05.621 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:05.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:06.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:06.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:06.953 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:38:06.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:38:06.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:06.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:07.295 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:07.749 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:07.745+0000 7fb2199927c0 -1 osd.1 0 log_to_monitors true 2024-06-11T18:38:08.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:08.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:08.297 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:38:08.298 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:38:08.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:08.298 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:08.766 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:09.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:09.767 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:09.768 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:38:09.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:38:09.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:09.768 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:10.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:11.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:11.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:11.228 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:38:11.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:38:11.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:11.228 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:11.667 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:12.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:12.669 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:12.669 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:38:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:38:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:12.670 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:13.132 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:14.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:14.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:14.135 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:38:14.135 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:38:14.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:14.136 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:14.473 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:15.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:15.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:15.476 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:38:15.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:38:15.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:15.476 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:15.812 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:16.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:16.814 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:16.815 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:38:16.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:38:16.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:16.815 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:17.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:18.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:18.166 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:18.166 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:38:18.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:38:18.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:18.167 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:18.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:19.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:19.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:19.508 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:38:19.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:38:19.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:19.508 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:20.024 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:21.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:21.026 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:38:21.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:38:21.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:21.027 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:21.363 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:22.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:22.365 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:22.366 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:38:22.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:38:22.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:22.366 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:23.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:23.701 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:23.702 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:38:23.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:38:23.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:23.702 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:25.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:25.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:25.102 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:38:25.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:38:25.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:25.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:26.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:26.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:26.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:38:26.523 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:38:26.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:26.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:26.863 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:27.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:27.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:27.865 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:38:27.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:38:27.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:27.866 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:28.281 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:29.286 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:29.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:29.287 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:38:29.287 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:38:29.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:29.288 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:29.992 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:30.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:30.994 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:30.995 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:38:30.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:38:30.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:30.995 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:31.445 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:32.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:32.447 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:32.448 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:38:32.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:38:32.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:32.448 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:33.101 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:34.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:34.103 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:34.104 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:38:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:38:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:34.104 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:34.874 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:35.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:35.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:35.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:38:35.877 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:38:35.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:35.877 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:36.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:37.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:37.697 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:37.697 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:38:37.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:38:37.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:37.698 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:38.194 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:39.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:39.196 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:39.197 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:38:39.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:38:39.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:39.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:40.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:41.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:41.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:41.146 INFO:tasks.workunit.client.0.smithi012.stdout:25 2024-06-11T18:38:41.146 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-11T18:38:41.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:41.147 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:38:41.481 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/596885007,v1:127.0.0.1:6811/596885007] [v2:127.0.0.1:6812/596885007,v1:127.0.0.1:6813/596885007] exists,up 30af0c36-d79a-4c73-aad5-b7733ed88642 2024-06-11T18:38:41.484 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-11T18:38:41.485 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:38:41.486 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:38:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:38:41.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-11T18:38:41.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:38:41.488 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-11T18:38:41.489 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-11T18:38:41.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7929dd2c-bf27-4d4d-9dda-22c560e68923 2024-06-11T18:38:41.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7929dd2c-bf27-4d4d-9dda-22c560e68923' 2024-06-11T18:38:41.489 INFO:tasks.workunit.client.0.smithi012.stdout:add osd2 7929dd2c-bf27-4d4d-9dda-22c560e68923 2024-06-11T18:38:41.489 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-11T18:38:41.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxmWhm5jFvHhAAJsOA0PAKoSpVLR8oBLkXqw== 2024-06-11T18:38:41.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxmWhm5jFvHhAAJsOA0PAKoSpVLR8oBLkXqw=="}' 2024-06-11T18:38:41.512 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7929dd2c-bf27-4d4d-9dda-22c560e68923 -i td/osd-scrub-repair/2/new.json 2024-06-11T18:38:41.940 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:38:41.958 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-11T18:38:41.959 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCxmWhm5jFvHhAAJsOA0PAKoSpVLR8oBLkXqw== --osd-uuid 7929dd2c-bf27-4d4d-9dda-22c560e68923 2024-06-11T18:38:41.995 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:41.993+0000 7f47e06647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:42.075 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.073+0000 7f47e06647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.073+0000 7f47e06647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:42.078 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.073+0000 7f47e06647c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:38:42.078 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.073+0000 7f47e06647c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:38:42.078 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.073+0000 7f47e06647c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-11T18:38:42.161 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:42.157+0000 7f47e06647c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-11T18:38:50.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-11T18:38:50.230 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-11T18:38:50.231 INFO:tasks.workunit.client.0.smithi012.stdout:adding osd2 key to auth repository 2024-06-11T18:38:50.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-11T18:38:50.232 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-11T18:38:50.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-11T18:38:50.643 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.2 2024-06-11T18:38:50.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:38:50.643 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-11T18:38:50.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-11T18:38:50.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-11T18:38:50.677 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:50.673+0000 7f78ad3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:50.713 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:50.709+0000 7f78ad3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:50.715 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:50.713+0000 7f78ad3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:38:51.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-11T18:38:51.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:38:51.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-11T18:38:51.008 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:38:51.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:38:51.009 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:51.009 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:38:51.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:38:51.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:51.010 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:51.277 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:51.273+0000 7f78ad3d37c0 -1 Falling back to public interface 2024-06-11T18:38:51.371 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:52.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:52.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:52.374 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:38:52.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:38:52.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:52.374 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:52.700 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:53.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:53.703 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:53.703 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:38:53.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:38:53.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:53.704 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:54.035 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:54.114 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:38:54.109+0000 7f78ad3d37c0 -1 osd.2 0 log_to_monitors true 2024-06-11T18:38:55.037 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:55.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:55.038 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:38:55.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:38:55.038 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:55.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:55.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:56.491 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:56.492 INFO:tasks.workunit.client.0.smithi012.stdout:4 2024-06-11T18:38:56.492 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-11T18:38:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:56.493 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:56.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:57.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:57.892 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:57.892 INFO:tasks.workunit.client.0.smithi012.stdout:5 2024-06-11T18:38:57.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-11T18:38:57.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:57.893 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:58.224 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:38:59.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:38:59.226 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:38:59.227 INFO:tasks.workunit.client.0.smithi012.stdout:6 2024-06-11T18:38:59.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-11T18:38:59.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:38:59.227 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:38:59.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:00.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:00.649 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:00.649 INFO:tasks.workunit.client.0.smithi012.stdout:7 2024-06-11T18:39:00.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-11T18:39:00.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:00.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:00.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:01.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:01.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:01.988 INFO:tasks.workunit.client.0.smithi012.stdout:8 2024-06-11T18:39:01.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-11T18:39:01.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:01.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:02.430 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:03.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:03.432 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:03.433 INFO:tasks.workunit.client.0.smithi012.stdout:9 2024-06-11T18:39:03.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-11T18:39:03.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:03.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:03.843 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:04.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:04.845 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:04.845 INFO:tasks.workunit.client.0.smithi012.stdout:10 2024-06-11T18:39:04.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-11T18:39:04.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:04.846 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:05.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:06.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:06.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:06.186 INFO:tasks.workunit.client.0.smithi012.stdout:11 2024-06-11T18:39:06.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-11T18:39:06.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:06.187 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:06.521 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:07.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:07.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:07.523 INFO:tasks.workunit.client.0.smithi012.stdout:12 2024-06-11T18:39:07.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-11T18:39:07.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:07.524 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:07.862 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:08.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:08.864 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:08.865 INFO:tasks.workunit.client.0.smithi012.stdout:13 2024-06-11T18:39:08.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-11T18:39:08.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:08.865 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:09.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:10.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:10.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:10.202 INFO:tasks.workunit.client.0.smithi012.stdout:14 2024-06-11T18:39:10.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-11T18:39:10.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:10.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:10.647 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:11.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:11.650 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:11.650 INFO:tasks.workunit.client.0.smithi012.stdout:15 2024-06-11T18:39:11.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-11T18:39:11.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:11.651 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:11.984 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:12.986 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:12.987 INFO:tasks.workunit.client.0.smithi012.stdout:16 2024-06-11T18:39:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-11T18:39:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:12.987 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:13.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:14.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-11T18:39:14.467 INFO:tasks.workunit.client.0.smithi012.stdout:17 2024-06-11T18:39:14.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:14.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:15.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:16.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:16.202 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:16.202 INFO:tasks.workunit.client.0.smithi012.stdout:18 2024-06-11T18:39:16.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-11T18:39:16.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:16.203 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:17.013 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:18.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:18.015 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:18.015 INFO:tasks.workunit.client.0.smithi012.stdout:19 2024-06-11T18:39:18.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-11T18:39:18.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:18.016 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:18.475 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:19.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:19.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:19.477 INFO:tasks.workunit.client.0.smithi012.stdout:20 2024-06-11T18:39:19.477 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-11T18:39:19.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:19.478 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:20.197 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:21.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:21.199 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:21.199 INFO:tasks.workunit.client.0.smithi012.stdout:21 2024-06-11T18:39:21.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-11T18:39:21.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:21.200 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:21.663 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:22.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:22.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:22.665 INFO:tasks.workunit.client.0.smithi012.stdout:22 2024-06-11T18:39:22.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-11T18:39:22.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:22.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:23.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:24.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:24.311 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:24.312 INFO:tasks.workunit.client.0.smithi012.stdout:23 2024-06-11T18:39:24.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-11T18:39:24.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:24.312 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:24.902 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:25.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:25.904 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:25.905 INFO:tasks.workunit.client.0.smithi012.stdout:24 2024-06-11T18:39:25.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-11T18:39:25.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:25.905 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:26.274 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:39:27.275 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:39:27.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:39:27.276 INFO:tasks.workunit.client.0.smithi012.stdout:25 2024-06-11T18:39:27.276 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-11T18:39:27.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:39:27.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-11T18:39:27.637 INFO:tasks.workunit.client.0.smithi012.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4072151696,v1:127.0.0.1:6819/4072151696] [v2:127.0.0.1:6820/4072151696,v1:127.0.0.1:6821/4072151696] exists,up 7929dd2c-bf27-4d4d-9dda-22c560e68923 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=testpool 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=false 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-06-11T18:39:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-11T18:39:28.105 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-06-11T18:39:28.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-06-11T18:39:28.939 INFO:tasks.workunit.client.0.smithi012.stderr:pool 'testpool' created 2024-06-11T18:39:28.965 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-11T18:39:29.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' false = true ']' 2024-06-11T18:39:29.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-06-11T18:39:29.967 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:39:29.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:39:29.968 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:39:29.968 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:39:29.969 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:39:29.969 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:39:29.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:39:29.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:39:29.970 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:39:30.112 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:39:30.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:39:30.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:39:30.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:39:30.113 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:39:30.113 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:39:30.511 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:39:30.664 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-06-11T18:39:30.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-06-11T18:39:30.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-06-11T18:39:30.665 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:39:30.665 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:39:30.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-06-11T18:39:30.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-06-11T18:39:30.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-06-11T18:39:30.900 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:39:30.901 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:39:31.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-06-11T18:39:31.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-06-11T18:39:31.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509450' 2024-06-11T18:39:31.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:39:31.051 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-06-11T18:39:31.051 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:39:31.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:39:31.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-06-11T18:39:31.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:39:31.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-06-11T18:39:31.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-11T18:39:31.056 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 21474836504 2024-06-11T18:39:31.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:39:31.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-11T18:39:31.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:39:31.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:39:31.388 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-06-11T18:39:31.390 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:39:31.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-06-11T18:39:31.391 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:39:31.393 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-06-11T18:39:31.393 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 42949672978 2024-06-11T18:39:31.394 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-11T18:39:31.394 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:39:31.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-11T18:39:31.715 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:39:31.716 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-06-11T18:39:31.717 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:39:31.718 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:39:31.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-06-11T18:39:31.719 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:39:31.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-06-11T18:39:31.720 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-11T18:39:31.720 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509450 2024-06-11T18:39:31.721 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:39:32.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 9 -lt 64424509450 2024-06-11T18:39:32.089 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:39:33.091 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:39:33.092 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:39:33.428 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509450 2024-06-11T18:39:33.429 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:39:33.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:39:33.430 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:39:33.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:39:33.876 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:39:33.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:39:33.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:39:33.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:39:33.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:39:33.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:39:33.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:39:34.244 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:39:34.244 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:39:34.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:39:34.245 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-06-11T18:39:34.631 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-06-11T18:39:34.632 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-06-11T18:39:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:34.632 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:34.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:34.989 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:35.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:35.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:35.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:35.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:36.144 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:36.145 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:36.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:36.479 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:36.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:36.781 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:37.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:37.106 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:37.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:37.441 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:37.785 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:37.786 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:38.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:38.102 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:38.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:38.413 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:38.722 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:38.723 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:39.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:39.006 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:39.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:39.358 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:39.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:39.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:40.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:40.026 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:40.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:40.950 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:41.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:41.270 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:41.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:41.662 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:41.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:41.971 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:42.306 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:42.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:42.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:42.607 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:42.883 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:43.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:43.235 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:43.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:43.554 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:43.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:43.913 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:44.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-11T18:39:44.164 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-11T18:39:44.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-06-11T18:39:44.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-11T18:39:44.533 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-11T18:39:44.534 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-11T18:39:44.534 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T18:39:44.534 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T18:39:44.535 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T18:39:44.535 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:39:44.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-11T18:39:44.877 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-11T18:39:44.878 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-11T18:39:45.277 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-06-11T18:39:45.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-06-11T18:39:45.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-11T18:39:45.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-11T18:39:45.278 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-11T18:39:45.279 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-11T18:39:45.609 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-06-11T18:39:45.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-06-11T18:39:45.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-11T18:39:45.610 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-11T18:39:45.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-11T18:39:45.611 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-11T18:39:45.935 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-06-11T18:39:45.936 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-11T18:39:45.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:39:45.937 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:39:45.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:39:45.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:39:45.937 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:39:47.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:39:47.250 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-11T18:39:47.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:39:47.251 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:39:47.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:39:47.252 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:39:47.253 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:39:48.568 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:39:48.569 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-06-11T18:39:48.571 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:48.572 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:39:48.575 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:48.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:39:48.576 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-11T18:39:50.354 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-06-11T18:39:50.887 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:50.888 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-06-11T18:39:50.889 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:39:50.890 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-11T18:39:52.588 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:104778fc:::obj2:head# 2024-06-11T18:39:53.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:53.125 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-06-11T18:39:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:39:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-11T18:39:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:53.127 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:39:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:39:53.128 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-11T18:39:54.212 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-06-11T18:39:54.748 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:54.749 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:39:54.751 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-11T18:39:55.981 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-06-11T18:39:56.519 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:56.520 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-06-11T18:39:56.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:39:56.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-11T18:39:56.522 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:56.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:56.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:39:56.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:56.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:39:56.523 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-11T18:39:57.617 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:c52c9666:::obj5:head# 2024-06-11T18:39:58.156 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:58.157 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:39:58.160 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-11T18:39:59.087 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:233a42c1:::obj6:head# 2024-06-11T18:39:59.620 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:39:59.622 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:39:59.624 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:39:59.625 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-11T18:40:00.496 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:9a09113a:::obj7:head# 2024-06-11T18:40:01.033 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:01.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:01.036 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-11T18:40:02.007 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:a557efb1:::obj8:head# 2024-06-11T18:40:02.555 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:02.556 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:02.559 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-11T18:40:03.634 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:fe351e27:::obj9:head# 2024-06-11T18:40:04.168 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:04.170 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:04.172 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-11T18:40:05.185 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:d1337354:::obj10:head# 2024-06-11T18:40:05.721 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:05.722 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:05.724 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-11T18:40:06.622 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-06-11T18:40:07.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:07.656 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-11T18:40:08.559 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-06-11T18:40:09.597 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:09.598 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-06-11T18:40:09.599 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:09.600 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-11T18:40:10.403 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-06-11T18:40:10.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:10.941 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:10.943 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-11T18:40:11.797 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:e0a44829:::obj14:head# 2024-06-11T18:40:12.333 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:12.334 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:12.336 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-11T18:40:13.247 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:ab946124:::obj15:head# 2024-06-11T18:40:13.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:13.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:13.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-11T18:40:14.625 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-06-11T18:40:15.171 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:15.172 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-06-11T18:40:15.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:15.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-11T18:40:15.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:15.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:15.174 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:15.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:15.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:15.175 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-11T18:40:16.052 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-06-11T18:40:16.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:16.589 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-11T18:40:17.579 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:b9836a99:::obj18:head# 2024-06-11T18:40:18.116 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:18.117 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-06-11T18:40:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-11T18:40:18.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:18.120 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-11T18:40:18.956 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-06-11T18:40:19.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:19.497 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-06-11T18:40:19.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:19.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-11T18:40:19.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:19.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:19.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:19.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:19.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:19.500 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-11T18:40:20.399 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:24b5611b:::obj20:head# 2024-06-11T18:40:20.948 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:20.949 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-06-11T18:40:20.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:20.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-06-11T18:40:20.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:20.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:20.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:20.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:20.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:20.953 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-06-11T18:40:21.902 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-06-11T18:40:22.437 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:22.438 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-06-11T18:40:22.439 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:22.440 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-06-11T18:40:23.521 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:c0c49def:::obj22:head# 2024-06-11T18:40:24.057 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:24.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:24.060 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-06-11T18:40:24.981 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:242c40e0:::obj23:head# 2024-06-11T18:40:25.517 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:25.518 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:25.520 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-06-11T18:40:26.416 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:7eacede5:::obj24:head# 2024-06-11T18:40:26.952 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:26.954 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-11T18:40:26.956 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:26.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:26.957 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-06-11T18:40:27.927 INFO:tasks.workunit.client.0.smithi012.stdout:remove 0#1:7109478f:::obj25:head# 2024-06-11T18:40:28.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-11T18:40:28.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:28.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-06-11T18:40:29.504 INFO:tasks.workunit.client.0.smithi012.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:40:30.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:40:30.045 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:40:30.046 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:40:30.047 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:40:30.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:40:30.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:40:30.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:40:30.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:40:30.048 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-11T18:40:30.049 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.1 2024-06-11T18:40:30.049 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-11T18:40:30.050 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:40:30.050 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-11T18:40:30.051 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-11T18:40:30.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:40:30.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:40:30.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:40:30.086 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:30.084+0000 7faf5e5797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:30.089 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:30.084+0000 7faf5e5797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:30.092 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:30.088+0000 7faf5e5797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:30.385 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:30.386 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:40:30.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:40:30.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:30.387 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:40:30.648 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:30.644+0000 7faf5e5797c0 -1 Falling back to public interface 2024-06-11T18:40:30.743 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:31.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:31.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:31.745 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:40:31.745 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:40:31.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:31.746 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:40:32.072 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:33.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:33.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:33.074 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:40:33.074 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:40:33.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:33.075 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:40:33.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:33.583 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:33.580+0000 7faf5e5797c0 -1 osd.1 23 log_to_monitors true 2024-06-11T18:40:34.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:34.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:34.496 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:40:34.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:40:34.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:34.496 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-11T18:40:34.938 INFO:tasks.workunit.client.0.smithi012.stdout:osd.1 up in weight 1 up_from 27 up_thru 22 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/3445859294,v1:127.0.0.1:6803/3445859294] [v2:127.0.0.1:6804/3445859294,v1:127.0.0.1:6805/3445859294] exists,up 30af0c36-d79a-4c73-aad5-b7733ed88642 2024-06-11T18:40:34.940 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-11T18:40:34.941 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-11T18:40:34.942 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-11T18:40:34.944 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-11T18:40:34.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-11T18:40:34.946 INFO:tasks.workunit.client.0.smithi012.stdout:start osd.0 2024-06-11T18:40:34.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=eb67d097-ff25-4800-88fd-ffa4a93f986d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61904/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-11T18:40:34.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-11T18:40:34.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-11T18:40:34.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-11T18:40:34.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-11T18:40:34.947 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-11T18:40:34.972 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:34.968+0000 7f131b87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:35.062 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:35.060+0000 7f131b87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:35.064 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:35.060+0000 7f131b87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-11T18:40:35.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-11T18:40:35.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-11T18:40:35.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stdout:0 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:40:35.309 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:35.601 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:35.596+0000 7f131b87d7c0 -1 Falling back to public interface 2024-06-11T18:40:35.652 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:36.653 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:36.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:36.654 INFO:tasks.workunit.client.0.smithi012.stdout:1 2024-06-11T18:40:36.654 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-11T18:40:36.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:36.655 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:40:36.988 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:37.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:37.990 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:37.991 INFO:tasks.workunit.client.0.smithi012.stdout:2 2024-06-11T18:40:37.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-11T18:40:37.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:37.991 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:40:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-11T18:40:38.352 INFO:tasks.workunit.client.0.smithi012.stderr:2024-06-11T18:40:38.348+0000 7f131b87d7c0 -1 osd.0 21 log_to_monitors true 2024-06-11T18:40:39.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-11T18:40:39.327 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-11T18:40:39.328 INFO:tasks.workunit.client.0.smithi012.stdout:3 2024-06-11T18:40:39.328 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-11T18:40:39.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-11T18:40:39.329 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-11T18:40:39.944 INFO:tasks.workunit.client.0.smithi012.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/3905796428,v1:127.0.0.1:6811/3905796428] [v2:127.0.0.1:6812/3905796428,v1:127.0.0.1:6813/3905796428] exists,up 7d31479b-a2ab-4e34-828b-2ddf511fbfbf 2024-06-11T18:40:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-11T18:40:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-11T18:40:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-11T18:40:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-06-11T18:40:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:40:39.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:40:39.946 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:40:39.946 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:40:39.946 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:40:39.946 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:40:39.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:40:39.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:40:39.947 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:40:40.077 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:40:40.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:40:40.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:40:40.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:40:40.078 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:40:40.078 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:40:40.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:40:40.464 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:40:40.464 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:40:40.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:40:40.464 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:40.465 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:40:40.677 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018882 2024-06-11T18:40:40.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018882 2024-06-11T18:40:40.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882' 2024-06-11T18:40:40.678 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:40.678 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:40:40.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=115964116995 2024-06-11T18:40:40.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 115964116995 2024-06-11T18:40:40.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-06-11T18:40:40.871 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:40.872 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:40:41.029 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509465 2024-06-11T18:40:41.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509465 2024-06-11T18:40:41.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509465' 2024-06-11T18:40:41.030 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:41.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018882 2024-06-11T18:40:41.030 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:41.031 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:40:41.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018882 2024-06-11T18:40:41.032 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:41.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018882 2024-06-11T18:40:41.034 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-11T18:40:41.034 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 128849018882 2024-06-11T18:40:41.034 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:40:41.396 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-11T18:40:41.397 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:41.397 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-115964116995 2024-06-11T18:40:41.398 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:41.399 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:40:41.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-115964116995 2024-06-11T18:40:41.400 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:41.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=115964116995 2024-06-11T18:40:41.402 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-06-11T18:40:41.402 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 115964116995 2024-06-11T18:40:41.403 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:40:41.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 115964116995 -lt 115964116995 2024-06-11T18:40:41.753 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:41.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509465 2024-06-11T18:40:41.754 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:40:41.756 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509465 2024-06-11T18:40:41.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:41.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509465 2024-06-11T18:40:41.758 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-06-11T18:40:41.758 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509465 2024-06-11T18:40:41.759 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:40:42.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509464 -lt 64424509465 2024-06-11T18:40:42.119 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:40:43.121 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:40:43.122 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:40:43.489 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509466 -lt 64424509465 2024-06-11T18:40:43.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:40:43.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:40:43.491 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:40:43.930 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:40:43.931 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:40:43.931 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:40:43.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:40:43.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:40:43.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:40:43.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:40:43.932 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:40:44.247 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:40:44.248 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:40:44.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:40:44.249 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:40:44.640 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:40:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:40:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:40:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-06-11T18:40:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-06-11T18:40:44.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-06-11T18:40:44.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:44.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:44.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:44.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:44.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:44.999 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-06-11T18:40:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-11T18:40:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:40:45.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:40:45.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:45.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:45.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:45.316 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:45.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:39:28.754423+0000 '>' 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:45.644 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:40:46.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:40:46.646 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:40:46.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:40:46.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:46.647 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:46.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:46.648 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:47.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:39:28.754423+0000 '>' 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:47.093 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:40:48.094 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:40:48.095 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:40:48.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:40:48.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:48.096 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:48.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:48.097 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:39:28.754423+0000 '>' 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:48.431 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:40:49.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:40:49.433 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:40:49.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:40:49.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:49.434 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:49.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:49.435 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:49.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:39:28.754423+0000 '>' 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:49.770 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-11T18:40:50.771 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-11T18:40:50.772 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-11T18:40:50.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-11T18:40:50.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-11T18:40:50.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-11T18:40:50.773 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-11T18:40:50.774 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-11T18:40:45.644914+0000 '>' 2024-06-11T18:39:28.754423+0000 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-11T18:40:51.107 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-11T18:40:51.108 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-11T18:40:51.108 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-11T18:40:51.109 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-11T18:40:51.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-11T18:40:51.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-11T18:40:51.109 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-11T18:40:51.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-11T18:40:51.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-11T18:40:51.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-11T18:40:51.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-11T18:40:51.240 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:40:51.241 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:40:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:40:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:40:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:40:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:40:51.572 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:51.573 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:40:51.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018885 2024-06-11T18:40:51.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018885 2024-06-11T18:40:51.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885' 2024-06-11T18:40:51.757 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:51.757 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:40:51.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=115964116998 2024-06-11T18:40:51.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 115964116998 2024-06-11T18:40:51.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885 1-115964116998' 2024-06-11T18:40:51.901 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:51.902 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:40:52.039 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-06-11T18:40:52.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-06-11T18:40:52.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018885 1-115964116998 2-64424509469' 2024-06-11T18:40:52.040 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:52.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018885 2024-06-11T18:40:52.040 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:52.041 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:40:52.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018885 2024-06-11T18:40:52.042 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:52.043 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018885 2024-06-11T18:40:52.043 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 128849018885 2024-06-11T18:40:52.044 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-06-11T18:40:52.044 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:40:52.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018884 -lt 128849018885 2024-06-11T18:40:52.362 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:40:53.364 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:40:53.365 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:40:53.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018885 -lt 128849018885 2024-06-11T18:40:53.690 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:53.691 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:53.691 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-115964116998 2024-06-11T18:40:53.692 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:40:53.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-115964116998 2024-06-11T18:40:53.693 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:53.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=115964116998 2024-06-11T18:40:53.695 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-06-11T18:40:53.695 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 115964116998 2024-06-11T18:40:53.695 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:40:54.052 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 115964116998 -lt 115964116998 2024-06-11T18:40:54.053 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:54.053 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-06-11T18:40:54.054 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:54.055 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:40:54.056 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-06-11T18:40:54.057 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:54.058 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-06-11T18:40:54.058 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509469 2024-06-11T18:40:54.059 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-06-11T18:40:54.059 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:40:54.406 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-06-11T18:40:54.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-11T18:40:54.407 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:40:54.408 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:40:54.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-11T18:40:54.835 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-11T18:40:54.836 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-11T18:40:55.213 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-11T18:40:55.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-11T18:40:55.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-11T18:40:55.214 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-11T18:40:55.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-11T18:40:55.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-11T18:40:55.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-11T18:40:55.606 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-06-11T18:40:55.964 INFO:tasks.workunit.client.0.smithi012.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-11T18:40:55.964 INFO:tasks.workunit.client.0.smithi012.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-11T18:40:47.749882+0000 21'30 31:134 [1,0,2] 1 [1,0,2] 1 21'30 2024-06-11T18:40:45.644914+0000 21'30 2024-06-11T18:40:45.644914+0000 0 1 periodic scrub scheduled @ 2024-06-12T18:40:45.644914+0000 17 0 2024-06-11T18:40:55.964 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-06-11T18:40:55.964 INFO:tasks.workunit.client.0.smithi012.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-11T18:40:55.964 INFO:tasks.workunit.client.0.smithi012.stderr:dumped pgs 2024-06-11T18:40:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-06-11T18:40:55.979 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-11T18:40:55.980 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-11T18:40:56.307 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-11T18:40:56.307 INFO:tasks.workunit.client.0.smithi012.stderr:1 2024-06-11T18:40:56.307 INFO:tasks.workunit.client.0.smithi012.stderr:2' 2024-06-11T18:40:56.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-11T18:40:56.308 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:56.308 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-11T18:40:56.494 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018887 2024-06-11T18:40:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018887 2024-06-11T18:40:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887' 2024-06-11T18:40:56.495 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:56.495 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-11T18:40:56.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=115964117000 2024-06-11T18:40:56.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 115964117000 2024-06-11T18:40:56.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-115964117000' 2024-06-11T18:40:56.641 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-11T18:40:56.642 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-11T18:40:56.787 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509470 2024-06-11T18:40:56.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509470 2024-06-11T18:40:56.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-115964117000 2-64424509470' 2024-06-11T18:40:56.788 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:56.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018887 2024-06-11T18:40:56.788 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:56.789 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-11T18:40:56.790 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018887 2024-06-11T18:40:56.791 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:56.792 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018887 2024-06-11T18:40:56.792 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.0 seq 128849018887 2024-06-11T18:40:56.793 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-06-11T18:40:56.793 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:40:57.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018886 -lt 128849018887 2024-06-11T18:40:57.124 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-11T18:40:58.125 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-11T18:40:58.126 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-11T18:40:58.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018887 2024-06-11T18:40:58.462 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:58.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-115964117000 2024-06-11T18:40:58.463 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:58.465 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-11T18:40:58.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-115964117000 2024-06-11T18:40:58.466 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:58.467 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=115964117000 2024-06-11T18:40:58.468 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.1 seq 115964117000 2024-06-11T18:40:58.468 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 115964117000' 2024-06-11T18:40:58.468 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-11T18:40:58.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 115964117000 -lt 115964117000 2024-06-11T18:40:58.824 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-11T18:40:58.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509470 2024-06-11T18:40:58.826 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-11T18:40:58.828 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-11T18:40:58.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509470 2024-06-11T18:40:58.829 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-11T18:40:58.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509470 2024-06-11T18:40:58.830 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-06-11T18:40:58.831 INFO:tasks.workunit.client.0.smithi012.stdout:waiting osd.2 seq 64424509470 2024-06-11T18:40:58.831 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-11T18:40:59.219 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509470 2024-06-11T18:40:59.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-11T18:40:59.220 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-06-11T18:40:59.376 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes": 210, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects": 30, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_object_clones": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_object_copies": 90, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_missing_on_primary": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_missing": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_degraded": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_misplaced": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_unfound": 0, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_dirty": 30, 2024-06-11T18:40:59.377 INFO:tasks.workunit.client.0.smithi012.stdout: "num_whiteouts": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_read": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_read_kb": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_write": 30, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_write_kb": 30, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_scrub_errors": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shallow_scrub_errors": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_deep_scrub_errors": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_recovered": 26, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes_recovered": 182, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_keys_recovered": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_omap": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_hit_set_archive": 0, 2024-06-11T18:40:59.378 INFO:tasks.workunit.client.0.smithi012.stdout: "num_bytes_hit_set_archive": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_kb": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_kb": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_promote": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_mode_high": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_flush_mode_low": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_mode_some": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_evict_mode_full": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_pinned": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_legacy_snapsets": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_large_omap_objects": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_manifest": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_omap_bytes": 0, 2024-06-11T18:40:59.379 INFO:tasks.workunit.client.0.smithi012.stdout: "num_omap_keys": 0, 2024-06-11T18:40:59.380 INFO:tasks.workunit.client.0.smithi012.stdout: "num_objects_repaired": 26 2024-06-11T18:40:59.380 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:40:59.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-11T18:40:59.380 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-06-11T18:40:59.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-06-11T18:40:59.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-06-11T18:40:59.528 INFO:tasks.workunit.client.0.smithi012.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-06-11T18:40:59.530 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-06-11T18:40:59.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-11T18:40:59.530 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-06-11T18:40:59.531 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-06-11T18:40:59.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-06-11T18:40:59.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-11T18:40:59.532 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-11T18:40:59.834 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout: "osd": 0, 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout: "up_from": 30, 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout: "seq": 128849018888, 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_pgs": 1, 2024-06-11T18:40:59.851 INFO:tasks.workunit.client.0.smithi012.stdout: "num_osds": 1, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_osds": 1, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "num_per_pool_omap_osds": 1, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb": 104857600, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used": 28480, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_data": 376, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_omap": 4, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_used_meta": 28091, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "kb_avail": 104829120, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "statfs": { 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "total": 107374182400, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "available": 107345018880, 2024-06-11T18:40:59.852 INFO:tasks.workunit.client.0.smithi012.stdout: "internally_reserved": 0, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "allocated": 385024, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "data_stored": 206078, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed": 0, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_allocated": 0, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "data_compressed_original": 0, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "omap_allocated": 4402, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "internal_metadata": 28765902 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "hb_peers": [ 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: 1, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: 2 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "snap_trim_queue_len": 0, 2024-06-11T18:40:59.853 INFO:tasks.workunit.client.0.smithi012.stdout: "num_snap_trimming": 0, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "num_shards_repaired": 26, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "op_queue_age_hist": { 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "histogram": [], 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "upper_bound": 1 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "perf_stat": { 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ms": 0, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ms": 0, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "commit_latency_ns": 0, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "apply_latency_ns": 0 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout: "alerts": [] 2024-06-11T18:40:59.854 INFO:tasks.workunit.client.0.smithi012.stdout:} 2024-06-11T18:40:59.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-11T18:40:59.855 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-11T18:41:00.163 INFO:tasks.workunit.client.0.smithi012.stderr:dumped all 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: main: code=1 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-scrub-repair 1 2024-06-11T18:41:00.183 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-11T18:41:00.184 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-11T18:41:00.185 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-11T18:41:00.497 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-11T18:41:00.498 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-11T18:41:00.499 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-11T18:41:00.500 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-11T18:41:00.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-11T18:41:00.502 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-11T18:41:00.503 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-11T18:41:00.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:41:00.504 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-11T18:41:00.505 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-11T18:41:00.505 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T18:41:00.507 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:41:00.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-06-11T18:41:00.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-06-11T18:41:00.509 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-06-11T18:41:00.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-06-11T18:41:00.511 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-11T18:41:00.526 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-11T18:41:00.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-11T18:41:00.527 INFO:tasks.workunit.client.0.smithi012.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61904 2024-06-11T18:41:00.527 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61904 2024-06-11T18:41:00.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-11T18:41:00.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-11T18:41:00.528 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 1 2024-06-11T18:41:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T18:41:00.530 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-06-11T18:41:00.530 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-11T18:41:00.882 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-11T18:41:01.078 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=783e8acd1229460bb96bdc8c2ac2140d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-11T18:41:01.081 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-11T18:41:01.090 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-11T18:41:01.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-11T18:41:01.132 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-11T18:41:01.141 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-11T18:41:01.141 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout:+hv01.front.sepi 67.205.162.81 3 u 96 512 377 0.178 +0.064 0.124 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout:+hv02.front.sepi 155.248.196.28 3 u 90 512 377 0.176 +1.415 0.286 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout:*hv03.front.sepi 74.6.168.72 3 u 230 256 377 0.186 +1.327 0.281 2024-06-11T18:41:01.165 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-11T18:41:01.166 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-11T18:41:01.175 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-11T18:41:01.175 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-11T18:41:01.184 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-11T18:41:01.194 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-11T18:41:01.204 INFO:teuthology.task.internal:Duration was 6761.415953 seconds 2024-06-11T18:41:01.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-11T18:41:01.214 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-11T18:41:01.214 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-11T18:41:01.258 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-11T18:41:01.258 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-06-11T18:41:01.259 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-11T18:41:01.315 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-11T18:41:01.315 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-11T18:41:01.393 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-11T18:41:01.393 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-11T18:41:01.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-11T18:41:01.495 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-11T18:41:01.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-11T18:41:01.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-11T18:41:01.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:41:01.559 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-06-11T18:41:01.600 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-11T18:41:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T18:41:01.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-11T18:41:01.655 INFO:teuthology.task.internal:Transferring archived files... 2024-06-11T18:41:01.657 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749704/remote/smithi012 2024-06-11T18:41:01.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-11T18:41:02.744 INFO:teuthology.task.internal:Removing archive directory... 2024-06-11T18:41:02.744 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-11T18:41:02.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-11T18:41:02.804 INFO:teuthology.task.internal:Not uploading archives. 2024-06-11T18:41:02.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-11T18:41:02.821 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-11T18:41:02.821 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-11T18:41:02.839 INFO:teuthology.orchestra.run.smithi012.stdout: 658625 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 11 18:41 /home/ubuntu/cephtest 2024-06-11T18:41:02.840 INFO:teuthology.orchestra.run.smithi012.stdout: 671758 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 11 16:58 /home/ubuntu/cephtest/mnt.0 2024-06-11T18:41:02.840 INFO:teuthology.orchestra.run.smithi012.stdout: 673784 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 11 17:10 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-11T18:41:02.840 INFO:teuthology.orchestra.run.smithi012.stdout: 789242 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 11 17:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-11T18:41:02.840 INFO:teuthology.orchestra.run.smithi012.stdout: 789243 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 11 18:41 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-06-11T18:41:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T18:41:02.841 INFO:teuthology.orchestra.run.smithi012.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-11T18:41:02.842 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-11T18:41:02.847 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-11T18:41:02.858 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-11T18:41:02.874 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi012 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-11T18:41:02.877 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 6761.415952682495 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi012 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_lflores@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=783e8acd1229460bb96bdc8c2ac2140d status: fail success: false 2024-06-11T18:41:02.878 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-11T18:41:02.976 INFO:teuthology.run:FAIL